Labels Annotations

​

Labels

groups:
  - name: node
    rules:
      - alert: NodeDown
        expr: up{job="node"} == 0
        labels:
          severity: warning
      - alert: MultipleNodesDown
        expr: avg_without(instance)(up{job="node"}) <= 0.5
        labels:
          severity: critical

​

Annotations

• {{ .Labels }} — the full labels map
• {{ .Labels.instance }} — the instance label value
• {{ .Value }} — the firing sample value (the metric value that caused the alert to fire)

groups:
  - name: node
    rules:
      - alert: node_filesystem_free_percent
        expr: 100 * node_filesystem_free_bytes{job="node"} / node_filesystem_size_bytes{job="node"} < 70
        annotations:
          description: "filesystem {{ .Labels.device }} on {{ .Labels.instance }} is low on space, current available space is {{ .Value }}"

​

Labels vs Annotations — Quick Comparison

​

Best Practices

• Keep labels stable and meaningful (e.g., severity, team, service) so routing rules remain predictable.
• Avoid putting large or frequently-changing text in labels — use annotations for verbose or dynamic content.
• Template annotations to include contextual details (instance, device, metric value) so notifications are actionable.
• Test templates with sample alerts to ensure correct rendering and escaping.

​

Links and References

• Alertmanager — Prometheus Alerting
• Prometheus Alerting Rules
• Go text/template package
• MDX Documentation