Skip to main content
We collected metrics with Prometheus. But metrics stored in a database don’t help anyone at three a.m. This lesson closes the loop: it turns thresholds into automated notifications. By the end you’ll understand the two-component alerting model, how to write alerting rules, how to configure Alertmanager routing, and how to keep notification noise under control. Real-world example In a fintech startup, Prometheus scrapes 200 targets and Grafana hosts 40+ dashboards. Over a weekend, Redis lag grew to two hours. The metric existed, but no alerting rule had been written. By Monday, 12,000 transactions were processed with stale exchange rates and a lot of money was lost.
An illustrated infographic shows people working around a giant lightbulb on a laptop screen with coins and charts. Text highlights "12,000 transactions to be processed with stale exchange rates" and sidebar metrics like "200 Targets" and "40 Dashboards."
A five-line alert would have caught this within minutes. Many teams stop at collection and dashboards. They build pretty Grafana screens and hope someone is watching — the “dashboard-only” approach leads to discovering outages from customer complaints.
An illustration of four people working around a large laptop screen that displays a giant lightbulb, bar chart, rising arrow and floating dollar coins. Below the graphic is the headline "The Fix?" with smaller explanatory text.
What you actually need is automated evaluation and smart routing:
  • Prometheus evaluates PromQL expressions continuously (many teams use a 15s evaluation interval — this is configurable and may differ from scrape intervals).
  • Alertmanager groups alerts, routes them based on labels, and delivers notifications to the right receivers (Slack, PagerDuty, email, etc.).
  • Noise-control mechanisms (silences, inhibition, grouping) keep notifications actionable so the team doesn’t ignore them.
A slide titled "Dashboards Don't Page You at 3 am" contrasting a "Dashboard-only approach" (collect metrics into Prometheus, build Grafana dashboards, hope someone watches, discover outages from customer complaints) with "What you actually need": automated evaluation, smart routing, and noise control (with brief descriptions). Copyright KodeKloud appears at the bottom.
Analogy: metrics without alerting are like a smoke detector with no alarm. Alerting model overview
  • Prometheus evaluates alerting rules: expressions written in PromQL, evaluated at the configured evaluation interval.
  • Alert states: inactive (resolved), pending, firing. The pending state is a debounce window to prevent transient spikes from firing alerts.
  • Alertmanager takes over once alerts are firing: it groups related alerts, routes them according to labels, and sends notifications to configured receivers.
In short: Prometheus decides when an alert should be considered firing; Alertmanager decides who gets notified and how. Writing alerting rules A typical alert rule contains five important fields: Example alert rule (Prometheus recording/alerting rules file):
Use for durations typically between a couple of minutes and ten minutes depending on your service’s tolerance for brief errors versus rapid notification. Too short → false positives; too long → slow detection.
How Alertmanager handles alerts (routing and grouping) Alertmanager uses a first-match routing tree. Order matters — top-most, most specific routes should come first. Key routing concepts:
  • route.receiver: default receiver if no child route matches.
  • group_by: labels used to group alerts into a single notification (e.g., ["alertname", "namespace"]).
  • group_wait: time to wait before sending the first notification for a group (commonly ~30s).
  • group_interval: minimum time between notifications for the same group.
  • routes: child routes with match conditions evaluated in order; the first matching child route is used.
Example Alertmanager route configuration:
Order your Alertmanager routes from most specific/critical to least. Put PagerDuty or on-call receivers near the top so critical alerts match first.
Why grouping and ordering matter
  • Grouping reduces noise by combining many similar alerts into one notification.
  • Proper ordering ensures urgent alerts reach the right people immediately, and broader routes catch anything else.
Controlling alert noise If teams receive hundreds of notifications daily they will start ignoring them. Use these mechanisms to keep alerts actionable:
  • Silences: temporary mutes for alerts (set via Alertmanager UI or API). Good for planned maintenance windows.
  • Inhibition: automatically suppress lower-priority alerts when a higher-priority alert is firing (e.g., inhibit pod alerts when a node is down).
  • Group tuning: adjust group_by to control bundling (e.g., group by alertname for broad messages, add namespace or instance for more detail).
  • Review and prune: remove stale or noisy alerts during periodic reviews (quarterly is a reasonable cadence).
A slide titled "Controlling Alert Noise" showing three colored panels — Silences, Inhibition Rules, and Grouping Tuning — each with brief "What?/When?/How?" descriptions for managing monitoring alerts. It outlines ways to mute, auto-suppress, and group alerts to reduce notification noise.
Alert lifecycle (trace)
  1. Rule evaluation: Prometheus evaluates the PromQL expression.
  2. Pending: condition is true but for window has not elapsed; if it resolves during this time, no alert fires.
  3. Firing: condition remains true for the for duration; alert is marked firing and sent to Alertmanager.
  4. Routing & grouping: Alertmanager matches and groups alerts, then sends notifications to configured receivers.
  5. Resolution: when the condition clears, Alertmanager sends a resolved notification.
Best practices
  • Alert on symptoms, not causes. For example, prefer alerts on increased latency or elevated 5xx rates rather than raw CPU usage.
  • Include runbook URLs in annotations so on-call engineers get immediate, actionable steps.
  • Choose for durations appropriate for your environment (commonly 2–10 minutes).
  • Regularly review alert rules and routing configuration to remove stale or noisy alerts.
  • Keep critical routes first in Alertmanager’s routing tree to ensure proper delivery.
  • Use inhibition to suppress downstream noise during major incidents (e.g., node down → inhibit pod/instance alerts).
Recap
  • Alerting is a two-component model: Prometheus evaluates rules; Alertmanager routes and delivers notifications.
  • Alert rules need expressions, for debounce durations, labels for routing, and annotations (summary, description, runbook).
  • Alertmanager routing is first-match; order routes from most specific/critical to least.
  • Use silences, inhibition, and grouping tuning to reduce alert fatigue and keep notifications actionable.
Links and references

Watch Video

Practice Lab