Recording Rules

In this lesson we cover Prometheus recording rules — how they work, why to use them, and best practices for organizing and deploying them. Recording rules let Prometheus periodically evaluate a PromQL expression and store the result as a new time series in the TSDB. By precomputing expensive or frequently-used expressions, dashboards (Grafana or other tools) and alerts can query the stored series directly, which reduces CPU usage and query latency on Prometheus.

A presentation slide titled "Recording Rules" that explains how Prometheus periodically evaluates PromQL expressions and stores the resulting time series. It includes a diagram with server, recording rules, Prometheus and Grafana icons and arrows showing data flow and benefits like faster dashboards.

Configuration overview

Define recording rules in rule files and list them under rule_files: in prometheus.yml.
Prometheus evaluates each rule at its evaluation interval and stores the result as a new series whose metric name is the record value.
rule_files accepts globs (for example, /etc/prometheus/rules/*.yml) so you can split rules across multiple files.
After editing rule files, Prometheus does not automatically pick them up unless you reload or restart Prometheus. Use the HTTP reload endpoint to avoid a full restart when possible.

After updating rule files, make sure to either reload Prometheus (POST /-/reload) or restart the service. Failing to reload means Prometheus continues using the old rules.

Example prometheus.yml (global intervals and rule_files):

global:
  scrape_interval: 15s        # how often to scrape targets
  evaluation_interval: 15s    # how often to evaluate rules

rule_files:
  - 'rules.yml'               # load this file; globs like 'rules/*.yml' also work

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "node"
    static_configs:
      - targets: ["192.168.1.168:9100", "192.168.1.168:9200"]

  - job_name: "api"
    static_configs:
      - targets: ["192.168.1.168:8000"]

Rule file structure

A rule file contains one or more rule groups under the groups: key. Each group may define:

name: identifier for the group.
interval (optional): overrides the global evaluation_interval for rules in that group.
rules: list of recording or alerting rules.

Evaluation semantics:

Groups are evaluated in parallel.
Rules within the same group are evaluated sequentially in the order declared. This allows later rules in a group to reference metrics recorded by earlier rules during the same evaluation run.

A diagram titled "Rules" showing nested Group2 and Group1 panels containing Rule1, Rule2, and Rule3. Labels indicate "sequential" and "parallel" grouping, with an arrow from the groups to a cylindrical database icon.

Rule fields

Field	Description	Example
`record`	Metric name for the newly stored series. Follow a structured naming convention.	`node_filesystem_free_percent`
`expr`	PromQL expression to evaluate.	`100 * node_filesystem_free_bytes{job="node"} / node_filesystem_size_bytes{job="node"}`
`labels`	(Optional) Add or overwrite labels on the resulting series.	`labels: { env: "prod" }`
`annotations`	Not applicable for recording rules (used for alerts).	N/A

PromQL examples (raw expressions)

Calculate percentage of used memory:

100 - (100 * node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)

Note: node_memory_MemAvailable_bytes is generally a better estimate of available memory than node_memory_MemFree_bytes. If your exporter doesn’t expose MemAvailable, you can fall back to MemFree, but it may underestimate actual available memory. Calculate percentage of free space on filesystems:

100 * node_filesystem_free_bytes / node_filesystem_size_bytes

Recording rule example

This sample rule group computes per-filesystem free percent, then produces an instance-level average. The record name becomes a new metric you can query directly.

groups:
  - name: example1
    interval: 15s
    rules:
      - record: node_filesystem_free_percent
        expr: 100 * node_filesystem_free_bytes{job="node"} / node_filesystem_size_bytes{job="node"}

      - record: node_filesystem_free_percent_avg
        expr: avg by(instance) (node_filesystem_free_percent)

Because rules in a group are evaluated in order, the second rule can reference the metric produced by the first rule in the same evaluation run.

Best practices for naming recording rules

Use a structured name to indicate the aggregation level, the base metric, and operations performed. A common pattern is: level:metric_name:operations

level: label set left after aggregation (e.g., job, job_method)
metric_name: base metric or subject (e.g., http_errors)
operations: transformations (e.g., rate5m, sum, avg)

Example: create a 5m rate of http_errors aggregated by job, method, and path:

- record: job_method_path:http_errors:rate5m
  expr: sum without (instance) (rate(http_errors{job="api"}[5m]))

If you remove path from aggregation, change level to job_method. Remove method too and level becomes job.

Grouping rules by job

Keep rules for a specific job together in one group. This simplifies management and makes sequential dependencies explicit. Example structure:

groups:
  - name: node          # all rules for job="node"
    interval: 15s
    rules:
      - record: node_memory_memFree_percent
        expr: 100 - (100 * node_memory_MemFree_bytes{job="node"} / node_memory_MemTotal_bytes{job="node"})

  - name: docker        # all rules for job="docker"
    interval: 15s
    rules:
      # docker related rules go here

Deploying rule files

Create a rule file (e.g. /etc/prometheus/rules.yml) and add your groups: content.
Ensure prometheus.yml lists the file under rule_files.
Reload Prometheus config or restart Prometheus.

Commands:

Action	Command
Restart via systemd	`sudo systemctl restart prometheus`
Graceful reload (recommended)	`curl -X POST http://localhost:9090/-/reload`

Checking rules in the UI

Open the Prometheus expression browser and visit Status → Rules. You will see rule groups and the evaluation status (OK, Error, etc.). You can also query recorded metrics by their record name in the expression browser or in Grafana. Documentation reference: Prometheus recording rules and management

Example rule definitions (quick reference)

record: node_filesystem_free_percent
expr: 100 * node_filesystem_free_bytes{job="node"} / node_filesystem_size_bytes{job="node"}

record: node_filesystem_free_percent_avg
expr: avg by(instance) (node_filesystem_free_percent)

Example query results (recorded metrics)

node_filesystem_free_percent{device="/dev/sda2", fstype="vfat", instance="192.168.1.168:9100", job="node", mountpoint="/boot/efi"} 98.97682793770934
node_filesystem_free_percent{device="/dev/sda3", fstype="ext4", instance="192.168.1.168:9100", job="node", mountpoint="/"} 16.005607045046514
node_filesystem_free_percent{device="/dev/sda3", fstype="ext4", instance="192.168.1.168:9100", job="node", mountpoint="/var/snap/firefox/common/host-hunspell"} 16.005607045046514
node_filesystem_free_percent{device="tmpfs", fstype="tmpfs", instance="192.168.1.168:9100", job="node", mountpoint="/run"} 99.79180265142136
node_filesystem_free_percent{device="tmpfs", fstype="tmpfs", instance="192.168.1.168:9100", job="node", mountpoint="/run/lock"} 99.921875
node_filesystem_free_percent{device="tmpfs", fstype="tmpfs", instance="192.168.1.168:9100", job="node", mountpoint="/run/snapd/ns"} 99.79180265142136
node_filesystem_free_percent{device="tmpfs", fstype="tmpfs", instance="192.168.1.168:9100", job="node", mountpoint="/run/user/1000"} 99.65394224493011

Use descriptive record names following the level:metric_name:operations pattern to keep metric namespaces consistent and understandable.
Prefer using the HTTP reload endpoint for production systems to avoid disrupting scraping and state where possible.
Keep rule groups small and focused per job so dependencies (sequential rules) are explicit and predictable.

Introduction

Observability Fundamentals

Prometheus Fundamentals

PromQL

Dashboarding & Visualization

Application Instrumentation

Service Discovery

Push Gateway

Alerting

Monitoring Kubernetes

Conclusion

Scaling & Long Term Storage

Configuration overview

Rule file structure

Rule fields

PromQL examples (raw expressions)

Recording rule example

Best practices for naming recording rules

Grouping rules by job

Deploying rule files

Checking rules in the UI

Example rule definitions (quick reference)

Example query results (recorded metrics)

Watch Video

Practice Lab

​Configuration overview

​Rule file structure

​Rule fields

​PromQL examples (raw expressions)

​Recording rule example

​Best practices for naming recording rules

​Grouping rules by job

​Deploying rule files

​Checking rules in the UI

​Example rule definitions (quick reference)

​Example query results (recorded metrics)

Watch Video

Practice Lab

Configuration overview

Rule file structure

Rule fields

PromQL examples (raw expressions)

Recording rule example

Best practices for naming recording rules

Grouping rules by job

Deploying rule files

Checking rules in the UI

Example rule definitions (quick reference)

Example query results (recorded metrics)