Alertmanager Rules

​

Basic AlertmanagerConfig example

apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: alert-config
  labels:
    resource: prometheus
spec:
  route:
    groupBy: ["severity"]
    groupWait: 30s
    groupInterval: 5m
    repeatInterval: 12h
    receiver: "webhook"
  receivers:
    - name: "webhook"
      webhookConfigs:
        - url: "http://example.com/"

• metadata.labels — used to match this object to Alertmanager instances (via alertmanagerConfigSelector).
• spec.route — route behavior (grouping and timing).
• spec.receivers — one or more receivers (here: webhook).

​

Why Alertmanager might not pick up your AlertmanagerConfig

spec:
  alertmanagerConfigNamespaceSelector: {}
  alertmanagerConfigSelector: {}

​

Important syntax differences: alertmanager.yml vs AlertmanagerConfig CRD

• Property naming: standalone alertmanager.yml uses snake_case (e.g., group_wait), while the CRD uses camelCase (e.g., groupWait).
• Label matchers: in alertmanager.yml you can write job: kubernetes. In the CRD matchers must be objects with name and value.

route:
  receiver: staff
  group_by: ['severity']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 12h
routes:
  - matchers:
      job: kubernetes
    receiver: infra
    group_by: ['severity']

spec:
  route:
    groupBy: ["alertname"]
    groupWait: 30s
    groupInterval: 5m
    repeatInterval: 12h
    receiver: "webhook"
  routes:
    - matchers:
        - name: job
          value: kubernetes
      receiver: "infra"
      groupBy: ["severity"]

​

Configure the Helm chart so Alertmanager finds AlertmanagerConfig objects

1. Retrieve the default values for the kube-prometheus-stack chart:

helm show values prometheus-community/kube-prometheus-stack > values.yaml

2. Edit values.yaml and set alertmanagerConfigSelector to match the labels you will use on AlertmanagerConfig objects. For example, to match resource: prometheus:

alertmanagerConfigSelector:
  matchLabels:
    resource: prometheus

3. Upgrade the Helm release using the modified values file:

helm upgrade prometheus prometheus-community/kube-prometheus-stack -f values.yaml

spec:
  alertmanagerConfigNamespaceSelector: {}
  alertmanagerConfigSelector:
    matchLabels:
      resource: prometheus

​

Create and apply an AlertmanagerConfig

apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: alert-config
  labels:
    resource: prometheus
spec:
  route:
    groupBy: ["severity"]
    groupWait: 30s
    groupInterval: 5m
    repeatInterval: 12h
    receiver: "webhook"
  receivers:
    - name: "webhook"
      webhookConfigs:
        - url: "http://example.com/"

kubectl apply -f alert.yaml
# Expected output:
# alertmanagerconfig.monitoring.coreos.com/alert-config created

kubectl get alertmanagerconfig
# NAME          AGE
# alert-config  17s

​

Verify the Alertmanager configuration in the UI

1. Find the Alertmanager service (commonly alertmanager-operated):

kubectl get svc

2. Port-forward to 9093:

kubectl port-forward service/alertmanager-operated 9093:9093

3. Open http://localhost:9093/ in your browser and inspect the “Status -> Configuration” page. You should see your webhook receiver and routing as part of the Alertmanager configuration generated from your AlertmanagerConfig CRD.

​

Troubleshooting tips

• If you see no AlertmanagerConfig objects in the Alertmanager UI but they exist in Kubernetes, re-check that:
  • The Helm chart alertmanagerConfigSelector matches the labels on your AlertmanagerConfig objects.
  • If you used namespace selectors, ensure alertmanagerConfigNamespaceSelector is set appropriately.
• Use kubectl get alertmanagers.monitoring.coreos.com -o yaml to inspect the Alertmanager CRD instance spec and verify selectors.
• Review the operator logs (Prometheus Operator) for errors about parsing AlertmanagerConfig objects.

• kube-prometheus-stack Helm chart: https://github.com/prometheus-community/helm-charts
• Prometheus Operator AlertmanagerConfig CRD: https://github.com/prometheus-operator/prometheus-operator
• Kubernetes basics: https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/