> ## Documentation Index
> Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Prometheus Alerting Rules AlertManager and Notification Routing

> Explains Prometheus alerting rules, Alertmanager routing, grouping, and noise control to turn metrics into actionable notifications with best practices and examples.

We collected metrics with Prometheus.

But metrics stored in a database don't help anyone at three a.m.

This lesson closes the loop: it turns thresholds into automated notifications. By the end you'll understand the two-component alerting model, how to write alerting rules, how to configure Alertmanager routing, and how to keep notification noise under control.

Real-world example

In a fintech startup, Prometheus scrapes 200 targets and Grafana hosts 40+ dashboards. Over a weekend, Redis lag grew to two hours. The metric existed, but no alerting rule had been written. By Monday, 12,000 transactions were processed with stale exchange rates and a lot of money was lost.

![An illustrated infographic shows people working around a giant lightbulb on a laptop screen with coins and charts. Text highlights "12,000 transactions to be processed with stale exchange rates" and sidebar metrics like "200 Targets" and "40 Dashboards."](https://mintcdn.com/kodekloud-c4ac6d9a/seqLTE3JlSrVAtw9/images/Prep-Course-Certified-Cloud-Native-Platform-Engineering-CNPE/Observability-and-Operations/Prometheus-Alerting-Rules-AlertManager-and-Notification-Routing/12000-transactions-stale-exchange-rates-infographic.jpg?fit=max\&auto=format\&n=seqLTE3JlSrVAtw9\&q=85\&s=e5d0e00914a4292b553f6b4c09a030b5)

A five-line alert would have caught this within minutes.

Many teams stop at collection and dashboards. They build pretty Grafana screens and hope someone is watching — the "dashboard-only" approach leads to discovering outages from customer complaints.

![An illustration of four people working around a large laptop screen that displays a giant lightbulb, bar chart, rising arrow and floating dollar coins. Below the graphic is the headline "The Fix?" with smaller explanatory text.](https://mintcdn.com/kodekloud-c4ac6d9a/seqLTE3JlSrVAtw9/images/Prep-Course-Certified-Cloud-Native-Platform-Engineering-CNPE/Observability-and-Operations/Prometheus-Alerting-Rules-AlertManager-and-Notification-Routing/team-laptop-lightbulb-growth-coins.jpg?fit=max\&auto=format\&n=seqLTE3JlSrVAtw9\&q=85\&s=b0aa49cf2cd62d49ae87d7c81a2ceda8)

What you actually need is automated evaluation and smart routing:

* Prometheus evaluates PromQL expressions continuously (many teams use a 15s evaluation interval — this is configurable and may differ from scrape intervals).
* Alertmanager groups alerts, routes them based on labels, and delivers notifications to the right receivers (Slack, PagerDuty, email, etc.).
* Noise-control mechanisms (silences, inhibition, grouping) keep notifications actionable so the team doesn’t ignore them.

![A slide titled "Dashboards Don't Page You at 3 am" contrasting a "Dashboard-only approach" (collect metrics into Prometheus, build Grafana dashboards, hope someone watches, discover outages from customer complaints) with "What you actually need": automated evaluation, smart routing, and noise control (with brief descriptions). Copyright KodeKloud appears at the bottom.](https://mintcdn.com/kodekloud-c4ac6d9a/seqLTE3JlSrVAtw9/images/Prep-Course-Certified-Cloud-Native-Platform-Engineering-CNPE/Observability-and-Operations/Prometheus-Alerting-Rules-AlertManager-and-Notification-Routing/dashboards-dont-page-you-3am.jpg?fit=max\&auto=format\&n=seqLTE3JlSrVAtw9\&q=85\&s=0ffb45c5b92155c913795c2c318bac82)

Analogy: metrics without alerting are like a smoke detector with no alarm.

Alerting model overview

* Prometheus evaluates alerting rules: expressions written in PromQL, evaluated at the configured evaluation interval.
* Alert states: `inactive` (resolved), `pending`, `firing`. The `pending` state is a debounce window to prevent transient spikes from firing alerts.
* Alertmanager takes over once alerts are `firing`: it groups related alerts, routes them according to labels, and sends notifications to configured receivers.

In short: Prometheus decides *when* an alert should be considered firing; Alertmanager decides *who* gets notified and *how*.

Writing alerting rules

A typical alert rule contains five important fields:

| Field | Purpose | Example |
| - | - | - |
| `alert` | Alert name; appears in Alertmanager and notifications | `HighErrorRate` |
| `expr` | PromQL expression that evaluates true/false or returns series | `rate(http_requests_total{code=~"5.."}[5m]) > 1` |
| `for` | Debounce duration; how long the condition must hold before firing | `5m` |
| `labels` | Metadata used for routing (e.g., severity, team) | `severity: critical` |
| `annotations` | Human-readable content: summary, description, runbook URL | `runbook: "https://runbook.example.com/high-5xx-rate"` |

Example alert rule (Prometheus recording/alerting rules file):

```yaml theme={null}
groups:
  - name: platform-alerts
    rules:
      - alert: HighErrorRate
        expr: |
          rate(http_requests_total{code=~"5.."}[5m]) > 1
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "High 5xx rate"
          description: "More than 1 5xx error per second over the last 5m."
          runbook: "https://runbook.example.com/high-5xx-rate"
```

Use `for` durations typically between a couple of minutes and ten minutes depending on your service's tolerance for brief errors versus rapid notification. Too short → false positives; too long → slow detection.

How Alertmanager handles alerts (routing and grouping)

Alertmanager uses a first-match routing tree. Order matters — top-most, most specific routes should come first.

Key routing concepts:

* `route.receiver`: default receiver if no child route matches.
* `group_by`: labels used to group alerts into a single notification (e.g., `["alertname", "namespace"]`).
* `group_wait`: time to wait before sending the first notification for a group (commonly \~30s).
* `group_interval`: minimum time between notifications for the same group.
* `routes`: child routes with match conditions evaluated in order; the first matching child route is used.

Example Alertmanager route configuration:

```yaml theme={null}
route:
  receiver: default-slack
  group_by: ["alertname", "namespace"]
  group_wait: 30s
  group_interval: 5m
  routes:
    - match:
        severity: critical
      receiver: pagerduty-oncall
    - match:
        team: payments
      receiver: payments-slack
```

Order your Alertmanager routes from most specific/critical to least. Put PagerDuty or on-call receivers near the top so critical alerts match first.

Why grouping and ordering matter

* Grouping reduces noise by combining many similar alerts into one notification.
* Proper ordering ensures urgent alerts reach the right people immediately, and broader routes catch anything else.

Controlling alert noise

If teams receive hundreds of notifications daily they will start ignoring them. Use these mechanisms to keep alerts actionable:

* Silences: temporary mutes for alerts (set via Alertmanager UI or API). Good for planned maintenance windows.
* Inhibition: automatically suppress lower-priority alerts when a higher-priority alert is firing (e.g., inhibit pod alerts when a node is down).
* Group tuning: adjust `group_by` to control bundling (e.g., group by `alertname` for broad messages, add `namespace` or `instance` for more detail).
* Review and prune: remove stale or noisy alerts during periodic reviews (quarterly is a reasonable cadence).

![A slide titled "Controlling Alert Noise" showing three colored panels — Silences, Inhibition Rules, and Grouping Tuning — each with brief "What?/When?/How?" descriptions for managing monitoring alerts. It outlines ways to mute, auto-suppress, and group alerts to reduce notification noise.](https://mintcdn.com/kodekloud-c4ac6d9a/seqLTE3JlSrVAtw9/images/Prep-Course-Certified-Cloud-Native-Platform-Engineering-CNPE/Observability-and-Operations/Prometheus-Alerting-Rules-AlertManager-and-Notification-Routing/controlling-alert-noise-silences-inhibition-grouping.jpg?fit=max\&auto=format\&n=seqLTE3JlSrVAtw9\&q=85\&s=59fc0fbc40ef5142ec03c171f2db6c05)

Alert lifecycle (trace)

1. Rule evaluation: Prometheus evaluates the PromQL expression.
2. Pending: condition is true but `for` window has not elapsed; if it resolves during this time, no alert fires.
3. Firing: condition remains true for the `for` duration; alert is marked `firing` and sent to Alertmanager.
4. Routing & grouping: Alertmanager matches and groups alerts, then sends notifications to configured receivers.
5. Resolution: when the condition clears, Alertmanager sends a resolved notification.

Best practices

* Alert on symptoms, not causes. For example, prefer alerts on increased latency or elevated 5xx rates rather than raw CPU usage.
* Include runbook URLs in annotations so on-call engineers get immediate, actionable steps.
* Choose `for` durations appropriate for your environment (commonly 2–10 minutes).
* Regularly review alert rules and routing configuration to remove stale or noisy alerts.
* Keep critical routes first in Alertmanager’s routing tree to ensure proper delivery.
* Use inhibition to suppress downstream noise during major incidents (e.g., node down → inhibit pod/instance alerts).

Recap

* Alerting is a two-component model: Prometheus evaluates rules; Alertmanager routes and delivers notifications.
* Alert rules need expressions, `for` debounce durations, labels for routing, and annotations (summary, description, runbook).
* Alertmanager routing is first-match; order routes from most specific/critical to least.
* Use silences, inhibition, and grouping tuning to reduce alert fatigue and keep notifications actionable.

Links and references

* Prometheus alerting rules: [https://prometheus.io/docs/prometheus/latest/configuration/alerting\_rules/](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/)
* Alertmanager configuration: [https://prometheus.io/docs/alerting/latest/alertmanager/](https://prometheus.io/docs/alerting/latest/alertmanager/)
* Best practices for alerting: [https://prometheus.io/docs/practices/alerting/](https://prometheus.io/docs/practices/alerting/)

> **[Watch Video](https://learn.kodekloud.com/user/courses/prep-course-certified-cloud-native-platform-engineer-cnpe/module/9bd090c8-8d99-4742-b50c-ae63e516e6b9/lesson/0cb3ce04-f9d4-4805-8980-70271d4cee9c)**

> **[Practice Lab](https://learn.kodekloud.com/user/courses/prep-course-certified-cloud-native-platform-engineer-cnpe/module/9bd090c8-8d99-4742-b50c-ae63e516e6b9/lesson/0e3f2975-c097-4dc6-8ac3-b6257be34389)**


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.