Labels Annotations

In this guide, we explore how to use labels and annotations in Alertmanager to enhance the context provided with your alerts. These features allow you to classify alerts effectively and add detailed descriptive information for clearer incident responses.

Labels: Classifying and Routing Alerts

Labels are essential for classifying and matching alerts in Alertmanager. By adding key-value pairs as labels in your alert rule configuration, you can filter and route alerts according to specific criteria. For example, the following configuration demonstrates two alert rules with different severity levels:

groups:
  - name: node
    rules:
      - alert: Node down
        expr: up{job="node"} == 0
        labels:
          severity: warning
      - alert: Multiple Nodes down
        expr: avg_without(instance)(up{job="node"}) <= 0.5
        labels:
          severity: critical

When you inspect the alerts in Prometheus, you will see the labels contributing to the alert details:

- name: Node down
  expr: up{job="node"} == 0
  for: 3m
  labels:
    severity: warning
- name: Multiple Nodes down
  expr: avg_without(instance)(up{job="node"}) <= 0.5
  labels:
    severity: critical

Annotations: Adding Descriptive Context

Annotations complement labels by providing additional descriptive details about an alert. They do not influence alert matching or routing but instead offer valuable context for understanding the alert's condition. Annotations use the Go templating language, enabling dynamic insertion of alert-related information.

For instance, you can access alert labels within an annotation using the following syntax:

• Use {{ .labels }} to retrieve all labels.
• Use {{ .labels.instance }} to get the value of the instance label.
• Use {{ .value }} to display the metric value at the time of the alert.

Consider an annotation named "description" that assembles information about the file system, device label, instance name, and current metric value. When triggered, the annotation might state: "file system /dev/sda3 on [target instance] is low on space. Current available space is 20.40345."

When alerts are sent via email or SMS, these descriptive annotations provide your team with the crucial context needed to understand and respond to the alert effectively.

Further Reading

• Kubernetes Documentation
• Prometheus Documentation
• Alerts and Alertmanager Setup

Understanding how to implement labels and annotations effectively in Alertmanager will streamline your incident response process and improve overall system monitoring.