Skip to main content
This guide walks through a concise demo: create Prometheus alerting rules, load them into Prometheus, and configure Alertmanager to group alerts and deliver notifications to Slack. It assumes Prometheus and Alertmanager are already installed and running. Prerequisites
  • Prometheus and Alertmanager installed and enabled as systemd services.
  • Basic familiarity with Prometheus metrics and labels.
Check the services:
Default ports:
  • Prometheus: 9090
  • Alertmanager: 9093
You can follow the official docs for installation and basic configuration:

​
1) Create alerting rules

Create the rules file at /etc/prometheus/rules.yaml. The example below defines one group my-alerts containing three alerts:
  • NodeDown — triggers when up{job="node"} goes to 0
  • DatabaseDown — triggers when up{job="db1"} goes to 0
  • DatabaseDown-dev — triggers when up{job="db2"} goes to 0
Each alert includes team and env labels to enable grouping and routing in Alertmanager. The annotations.message uses the Go template {{ .Labels.instance }} to show the instance address.
for: 0m moves alerts to firing immediately when the condition is true. To delay firing (reduce noise), use a longer duration such as for: 1m or for: 5m.
Save the file and restart Prometheus:

​
2) Update Prometheus configuration to load rules and scrape targets

Add the rule_files section to /etc/prometheus/prometheus.yml so Prometheus loads your alert rules. Ensure scrape_configs include jobs for node, db1, and db2. The example uses placeholder IPs — replace them with your actual targets.
Restart Prometheus after updating:
Open the Prometheus UI at http://localhost:9090 and check the Alerts page. Alerts will be shown in “not firing” (green) when targets are up; they move to pending/firing according to the for duration. Example active alert labels and annotations you will see in Prometheus when targets go down:
  • alertname=NodeDown env=prod instance=192.168.1.168:9100 job=node team=infra
  • alertname=DatabaseDown env=prod instance=192.168.1.168:9200 job=db1 team=database
  • alertname=DatabaseDown-dev env=dev instance=192.168.1.168:9300 job=db2 team=database

​
3) Configure Prometheus to forward alerts to Alertmanager

Add the following alerting block to /etc/prometheus/prometheus.yml so Prometheus knows where to send alerts:
Restart Prometheus:
Open Alertmanager at http://localhost:9093. Incoming alerts from Prometheus will appear there and will be grouped according to Alertmanager’s routing rules.

​
4) Configure Alertmanager routing and Slack receiver

Edit Alertmanager’s configuration at /etc/alertmanager/alertmanager.yml to define routes and receivers. The example below:
  • Matches alerts whose job label matches node, db1, or db2 (via match_re).
  • Groups alerts by team and env so each team/environment becomes a single grouped notification in Slack.
  • Sends grouped alerts to the slack receiver.
  • Uses a web.hook receiver as a fallback.
Replace api_url with your Slack incoming webhook URL and update channel as needed.
Notes on templates:
  • {{ .GroupLabels.team }} and {{ .GroupLabels.env }} reference the labels used for grouping.
  • The text template iterates over .Alerts and prints each alert’s annotations.message.
Do not commit or publish your Slack webhook URL publicly. Treat api_url like a secret and store it in a secure secret manager or environment-specific configuration.
Restart Alertmanager after saving the config:
After a short time, Alertmanager’s UI will show grouped alerts by team and env. For example:
  • env=dev + team=database
  • env=prod + team=database
  • env=prod + team=infra
If you filter in the Alertmanager UI by env="dev", only dev alerts will be displayed:
Screenshot of the Alertmanager web UI in a browser. It shows filtered alerts for env="dev", including "DatabaseDown" messages with instance IPs and controls to view info or silence alerts.

​
Slack notification example

When Alertmanager groups alerts and forwards them to Slack, a single Slack message is sent per group. Example messages that may appear in #alerts:
The title uses grouped labels to indicate which team and environment the notification concerns, and the message body lists each alert’s annotation message.

​
Files changed (quick reference)

​
Example alert summary table

​
Summary / Best practices

  • Define rules in /etc/prometheus/rules.yaml and load them via rule_files in prometheus.yml.
  • Use the for field to control pending → firing behavior and reduce alert flapping.
  • Point Prometheus to Alertmanager via the alerting section.
  • In Alertmanager:
    • Use routing (match, match_re) to direct alerts to the right receivers.
    • group_by the labels you want grouped (for example team and env) so one notification represents many alerts.
    • Use Go templates in receivers to build informative titles and messages.
  • Protect webhook URLs and sensitive receiver credentials; use secrets management for production.
Relevant references: The same routing and grouping concepts apply to other notification integrations such as PagerDuty, email, or custom webhooks — only the receiver configuration changes.

Watch Video