Explains Prometheus alerting, writing PromQL alert rules, alert lifecycles, and using Alertmanager to group, silence, route, and deliver notifications.
This lesson covers Prometheus alerting: how to detect issues using PromQL rules, how Prometheus forwards alerts, and how Alertmanager groups, silences, and delivers notifications (email, SMS, Slack, PagerDuty, etc.). Alerting ensures you receive timely notifications even when you are not actively watching dashboards — for example, when a production server runs low on disk at 3 a.m.
Prometheus alerting is driven by PromQL expressions. You write alerting rules that evaluate expressions; when an expression returns one or more result vectors, Prometheus treats each matching series as an alert.Example: alert when a filesystem has less than 1,000 bytes available. The Prometheus expression console might show:
Each result vector corresponds to a filesystem that matches the condition. If the query returns two vectors, Prometheus treats that as two separate alerts (one per matching series), each carrying the series labels.Prometheus only evaluates alerting rules and sends alert objects — it does not deliver notifications itself. Delivery (email, Slack, PagerDuty, SMS, etc.) is handled by Alertmanager.
Prometheus sends alert objects to Alertmanager. Alertmanager handles silencing, grouping, routing, and delivering notifications to various receivers such as email, Slack, or PagerDuty.
One Alertmanager instance can receive alerts from multiple Prometheus servers across environments. Alerting rules are grouped into rule groups (files), and you can place alerting rules alongside recording rules in the same file. The only syntax difference from a recording rule is using alert instead of record. You can also set a for clause on an alert to require the expression to be true continuously for a duration before the alert moves to firing — useful to avoid transient flaps.Example rule group that defines a recording rule for memory free percent and an alert that fires when that percent falls below 20%:
The recording rule computes the percent of free memory for targets with job="node".
The LowMemory alert uses that recorded metric and requires the condition to be true for 3 minutes (for: 3m) before transitioning to firing.
Another common pattern is alerting when a target is unreachable using the up metric. The for clause helps avoid false positives due to short-lived scrape failures:
Alert states and behavior are important to understand:
State
Meaning
Transition condition
Inactive
The alert expression currently returns no results (no matching series).
Expression returns no vectors.
Pending
The expression has returned one or more vectors, but the for duration (if set) has not yet elapsed.
Expression true, but for duration not yet satisfied.
Firing
The expression has been true for at least the for duration (or for omitted), and the alert is active — Prometheus has sent the alert to Alertmanager.
for duration elapsed while expression remains true (or no for set).
Use the for clause to reduce alert noise from short-lived network or scrape blips. If a node recovers before the for period elapses, the alert remains Pending and will not transition to Firing.
Best practices and quick tips:
Group recording and alerting rules logically in rule files and set a reasonable interval for evaluation to balance timeliness and load.
Use label-based grouping (in Alertmanager) to route alerts to the appropriate team or channel.
Test alert rules in a non-production environment and use silence windows for maintenance.
Add descriptive annotations to alerts to include runbook links and helpful context for responders.
