Adding Rules

This lesson shows how to add alerting and recording rules to Prometheus when using the Prometheus Operator. The Operator exposes a Custom Resource Definition (CRD) named PrometheusRule that lets you register Prometheus rules with a Prometheus instance. The CRD’s structure mirrors a standard Prometheus rules file, but it is stored as a Kubernetes custom resource.

PrometheusRule example

Below is an example PrometheusRule resource. It defines a rule group named api with an alert called down.

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: api-rules
  labels:
    release: prometheus
spec:
  groups:
    - name: api
      rules:
        - alert: down
          expr: up == 0
          for: 0m
          labels:
            severity: critical
          annotations:
            summary: Prometheus target missing {{$labels.instance}}

Notes on the example:

metadata.labels includes release: prometheus. Prometheus discovers PrometheusRule objects using the Prometheus resource’s ruleSelector (explained below), so these labels must match what your Prometheus instance is selecting.
spec.groups uses the same grouping construct as a standard Prometheus rules file — each group has a name and a set of rules.

How Prometheus finds rules

Prometheus uses a selector on the Prometheus custom resource to discover and register PrometheusRule objects dynamically. Check your Prometheus resource (CR) for the ruleSelector configuration. Example:

spec:
  ruleSelector:
    matchLabels:
      release: prometheus

Make sure your PrometheusRule objects include labels that match the ruleSelector on your Prometheus resource (for example, release: prometheus). Otherwise Prometheus will not pick them up.

ruleSelector only matches labels on the PrometheusRule resource. If your PrometheusRule is in a different namespace than your Prometheus CR, confirm your Operator installation supports cross-namespace rule discovery or place the rule in the same namespace as the Prometheus resource.

Key fields in a PrometheusRule

Field	Purpose	Example / Notes
`metadata.name`	Resource name in Kubernetes	`api-rules`
`metadata.labels`	Labels used by Prometheus `ruleSelector`	`release: prometheus`
`spec.groups`	Top-level grouping for rules (same as Prometheus file)	Each group has a `name` and `rules`
`spec.groups[*].rules`	Individual alerting/recording rules	Use `alert`, `expr`, `for`, `labels`, `annotations`

Steps to register rules

Create a rules file (for example rules.yaml) containing the PrometheusRule resource shown above.
Apply it to your cluster:

kubectl apply -f rules.yaml

Verify the resource exists:

kubectl get prometheusrules

You should see an entry such as api-rules. To confirm Prometheus has registered the rule, open the Prometheus UI and navigate to Status → Rules. You should see the api group and the down alert listed and in a healthy state.

Additional example rules

The following examples show multiple alerts grouped together. Add them to spec.groups[*].rules inside your PrometheusRule resource.

- alert: AlertmanagerFailedReload
  expr: max_over_time(alertmanager_config_last_reload_successful{job="prometheus-kube-prometheus-alertmanager",namespace="default"}[5m]) == 0
  for: 10m
  labels:
    severity: critical
  annotations:
    description: Configuration has failed to load for {{ $labels.namespace }}/{{ $labels.pod }}.
    runbook_url: https://runbooks.prometheus-operator.dev/runbooks/alertmanager/alertmanagerfailedreload
    summary: Reloading an Alertmanager configuration has failed.

- alert: AlertmanagerMembersInconsistent
  expr: max_over_time(alertmanager_cluster_members{job="prometheus-kube-prometheus-alertmanager",namespace="default"}[5m]) < on (namespace, service) group_left () count by (namespace, service) (max_over_time(alertmanager_cluster_members{job="prometheus-kube-prometheus-alertmanager",namespace="default"}[5m]))
  for: 15m
  labels:
    severity: critical

- alert: down
  expr: up == 0
  for: 0m
  labels:
    severity: critical
  annotations:
    summary: Prometheus target missing {{$labels.instance}}

Troubleshooting tips

If rules do not appear in the Prometheus UI, confirm the PrometheusRule labels match the ruleSelector on your Prometheus resource and that the rule resource is in the expected namespace.
Check the Prometheus Operator logs and the Prometheus server logs for any errors while loading rules.
Use kubectl describe prometheusrules <name> to inspect events or validation errors on the resource.

References

Prometheus Operator documentation: https://github.com/prometheus-operator/prometheus-operator
Prometheus rules and alerts: https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/

Introduction

Observability Fundamentals

Prometheus Fundamentals

PromQL

Dashboarding & Visualization

Application Instrumentation

Service Discovery

Push Gateway

Alerting

Monitoring Kubernetes

Conclusion

Scaling & Long Term Storage

PrometheusRule example

How Prometheus finds rules

Key fields in a PrometheusRule

Steps to register rules

Additional example rules

Troubleshooting tips

References

Watch Video

​PrometheusRule example

​How Prometheus finds rules

​Key fields in a PrometheusRule

​Steps to register rules

​Additional example rules

​Troubleshooting tips

​References

Watch Video

PrometheusRule example

How Prometheus finds rules

Key fields in a PrometheusRule

Steps to register rules

Additional example rules

Troubleshooting tips

References