Alertmanager Rules

This guide explains how to add Alertmanager routing and receiver configurations using the Prometheus Operator’s AlertmanagerConfig custom resource (CRD). AlertmanagerConfig objects let you register Alertmanager configuration snippets (routes, groupings, receivers) with an Alertmanager instance running in Kubernetes.

Example AlertmanagerConfig

This example demonstrates a simple route and a webhook receiver. Save it as a YAML file when creating the CRD.

apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: alert-config
  labels:
    resource: prometheus
spec:
  route:
    groupBy: ["severity"]
    groupWait: 30s
    groupInterval: 5m
    repeatInterval: 12h
    receiver: "webhook"
  receivers:
    - name: "webhook"
      webhookConfigs:
        - url: "http://example.com/"

This structure mirrors an Alertmanager configuration: a route, grouping settings, and one or more receivers. When deployed as a Kubernetes CRD, the AlertmanagerConfig will only be registered with an Alertmanager instance if that Alertmanager is configured to select it (selectors explained below).

Alertmanager selector settings you must know

The kube-prometheus-stack (and other Prometheus Operator installs) exposes two selector properties in the Alertmanager CR:

alertmanagerConfigSelector — controls which AlertmanagerConfig objects in the same namespace are selected.
alertmanagerConfigNamespaceSelector — controls which namespaces’ AlertmanagerConfig objects are selectable.

If these selectors are empty, Alertmanager will not pick up any AlertmanagerConfig resources. Inspect your Alertmanager CR with:

kubectl get alertmanagers.monitoring.coreos.com -o yaml

Typical default spec shows empty selectors:

spec:
  alertmanagerConfigNamespaceSelector: {}
  alertmanagerConfigSelector: {}
  externalUrl: http://prometheus-kube-prometheus-alertmanager.default:9093
  image: quay.io/prometheus/alertmanager:v0.24.0
  # ...

To enable automatic registration of AlertmanagerConfig objects, configure the Helm chart values so the Alertmanager’s alertmanagerConfigSelector matches the labels you add to your AlertmanagerConfig resources.

Key differences: Alertmanager YAML vs AlertmanagerConfig CRD

There are two important differences between a standalone alertmanager.yml and the AlertmanagerConfig CRD:

Area	Standalone `alertmanager.yml`	AlertmanagerConfig CRD
Field naming	snake_case (e.g., `group_wait`, `group_interval`)	camelCase (e.g., `groupWait`, `groupInterval`)
Matcher syntax	Simple key: value matchers under `match:`	Explicit `matchers` list with `name` and `value` fields

Examples for matcher and field naming differences: Standalone Alertmanager YAML (snake_case + simple matchers):

route:
  receiver: staff
  group_by: ['severity']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 12h
  routes:
    - match:
        job: kubernetes
      receiver: infra
      group_by: ['severity']

AlertmanagerConfig CRD (camelCase + explicit matchers):

spec:
  route:
    groupBy: ["alertname"]
    groupWait: 30s
    groupInterval: 5m
    repeatInterval: 12h
    receiver: "webhook"
    routes:
      - matchers:
          - name: job
            value: kubernetes
        receiver: "infra"
        groupBy: ["severity"]

When converting an existing Alertmanager alertmanager.yml to an AlertmanagerConfig CRD, convert snake_case fields to camelCase and convert simple label matchers into matchers objects with name and value.

Enable AlertmanagerConfig discovery in the Helm chart

Follow these steps to configure the kube-prometheus-stack Helm chart so Alertmanager automatically discovers AlertmanagerConfig resources.

Export the chart’s default values:

helm show values prometheus-community/kube-prometheus-stack > values.yaml

Edit values.yaml and add an alertmanagerConfigSelector so it matches the label you will use on your AlertmanagerConfig resources. For example, to match resource: prometheus:

alertmanagerConfigSelector:
  matchLabels:
    resource: prometheus

Note: If your chart organizes Alertmanager settings under a nested key (for example alertmanager:), place the selector under the appropriate key according to that chart’s values structure.

Upgrade the release with the modified values file:

helm upgrade prometheus prometheus-community/kube-prometheus-stack -f values.yaml

Verify the Alertmanager CR now contains the selector you configured:

kubectl get alertmanagers.monitoring.coreos.com -o yaml

You should see an updated spec like:

spec:
  alertmanagerConfigNamespaceSelector: {}
  alertmanagerConfigSelector:
    matchLabels:
      resource: prometheus
  externalUrl: http://prometheus-kube-prometheus-alertmanager.default:9093
  # ...

Create and apply an AlertmanagerConfig

Create a file named alert.yaml containing the AlertmanagerConfig example. Ensure the metadata.labels include the label the Alertmanager will select (in the example above, resource: prometheus):

apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: alert-config
  labels:
    resource: prometheus
spec:
  route:
    groupBy: ["severity"]
    groupWait: 30s
    groupInterval: 5m
    repeatInterval: 12h
    receiver: "webhook"
  receivers:
    - name: "webhook"
      webhookConfigs:
        - url: "http://example.com/"

Apply it:

kubectl apply -f alert.yaml

Verify creation:

kubectl get alertmanagerconfig
# Example output:
# NAME           AGE
# alert-config   17s

If you receive an error that the resource type does not exist, ensure the Prometheus Operator CRDs (including alertmanagerconfigs.monitoring.coreos.com) are installed in your cluster.

Verify Alertmanager picked up the config

Port-forward the Alertmanager service and open the UI to confirm the config was registered:

kubectl port-forward service/alertmanager-operated 9093:9093
# then open http://localhost:9093 and navigate to Status -> Config

On the Alertmanager Status -> Config page you should see the webhook receiver and the routes created via your AlertmanagerConfig CRD.

Troubleshooting tips

If your AlertmanagerConfig does not appear in Alertmanager:
- Confirm the Alertmanager’s alertmanagerConfigSelector matches the label on your AlertmanagerConfig.
- Confirm the namespace of the AlertmanagerConfig is selected by alertmanagerConfigNamespaceSelector if set.
- Confirm Prometheus Operator CRDs are installed and up to date.
Use kubectl describe on the Alertmanager and AlertmanagerConfig resources to see events and errors.
Check Alertmanager logs (via Pod logs) for registration errors.

Links and references

kube-prometheus-stack Helm chart: https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack
Prometheus Operator / AlertmanagerConfig CRD docs: https://github.com/prometheus-operator/prometheus-operator
Alertmanager documentation: https://prometheus.io/docs/alerting/latest/alertmanager/
Helm documentation: https://helm.sh/docs/
Kubernetes documentation: https://kubernetes.io/docs/

Introduction

Observability Fundamentals

Prometheus Fundamentals

PromQL

Dashboarding & Visualization

Application Instrumentation

Service Discovery

Push Gateway

Alerting

Monitoring Kubernetes

Conclusion

Scaling & Long Term Storage

Alertmanager Rules

Example AlertmanagerConfig

Alertmanager selector settings you must know

Key differences: Alertmanager YAML vs AlertmanagerConfig CRD

Enable AlertmanagerConfig discovery in the Helm chart

Create and apply an AlertmanagerConfig

Verify Alertmanager picked up the config

Troubleshooting tips

Links and references

Watch Video

Practice Lab

​Example AlertmanagerConfig

​Alertmanager selector settings you must know

​Key differences: Alertmanager YAML vs AlertmanagerConfig CRD

​Enable AlertmanagerConfig discovery in the Helm chart

​Create and apply an AlertmanagerConfig

​Verify Alertmanager picked up the config

​Troubleshooting tips

​Links and references

Watch Video

Practice Lab

Example AlertmanagerConfig

Alertmanager selector settings you must know

Key differences: Alertmanager YAML vs AlertmanagerConfig CRD

Enable AlertmanagerConfig discovery in the Helm chart

Create and apply an AlertmanagerConfig

Verify Alertmanager picked up the config

Troubleshooting tips

Links and references