Skip to main content
In this lesson we’ll cover Prometheus alerting: how to define alerting conditions with PromQL, how Prometheus generates alerts, and how Alertmanager handles routing and notifications. Alerting is critical for production systems — it ensures teams are notified of issues (disk full, node down, high latency) even when nobody is actively watching dashboards.
The image illustrates a system alert scenario, showing server icons with a "Low disk space" alert, and a sleeping administrator with the text suggesting a need for alert mechanisms when problems occur.
Prometheus alerting works by evaluating PromQL expressions. When an expression returns one or more vectors, Prometheus generates an alert instance for each matching timeseries. Example: alert when a filesystem has less than 1000 bytes available:
If this query returns a vector such as:
Prometheus creates one alert instance for that filesystem (because it has only 547 bytes available). If the query returns multiple vectors, each matching timeseries becomes a separate alert instance:
The second query produces two results, so Prometheus would produce two alert instances — one per filesystem.
Prometheus evaluates alert expressions and generates alert instances, but it does not deliver notifications (email, SMS, Slack). That responsibility belongs to Alertmanager, which receives alerts from Prometheus and handles deduplication, grouping, silencing, and routing to external channels.
The image illustrates the alerting process in Prometheus, showing that Prometheus triggers alerts but does not send notifications, which are handled by a separate process called Alertmanager. The diagram includes arrows showing alerts being managed by Alertmanager and distributed to users via platforms like Gmail, Slack, and another messaging service.

​
Alert rules and rule files

Alert rules are defined in the same rule format as recording rules. The main difference is the presence of the alert key for alerting rules, while recording rules use record. Both rule types can live in the same rule group. Example rule group containing a recording rule and an alert:
In this example:
  • node_memory_memFree_percent is a recording rule used to precompute a metric.
  • LowMemory is an alert that fires when node_memory_memFree_percent < 20 has held true for 3 minutes.
Another common alert is detecting targets that are down. The up metric returns 0 for unreachable targets:
The for clause delays the alert transition to firing until the expression has been true for the given duration. This prevents alerts from firing on transient scrape failures or brief network blips.
Use for to avoid false positives for sustained conditions (e.g., sustained high CPU). For extremely urgent conditions (e.g., data corruption, loss of control plane) keep for short or omit it to allow faster notifications.

​
Example alerts file

Prometheus reads rule files (commonly placed under /etc/prometheus/rules.yml or a rules directory). Example snippet:
When you open the Alerts tab in the Prometheus UI you’ll see these alerts and their current states.

​
Alert lifecycle and states

Prometheus alerts go through three states:

​
Quick reference: alert rule fields

​
Putting it together: Prometheus + Alertmanager

  • Prometheus evaluates alerting rules and creates alert instances when expressions match timeseries.
  • Prometheus sends alerts to one or more Alertmanager instances.
  • Alertmanager handles deduplication, grouping, silencing, and routing notifications to integrations such as email, Slack, PagerDuty, or webhooks.
  • A single Alertmanager can receive alerts from multiple Prometheus servers across environments.
This separation of concerns creates a reliable alerting pipeline: Prometheus for evaluation, Alertmanager for delivery and routing.

Watch Video