Recording Rules

Recording rules let Prometheus periodically evaluate a PromQL expression and persist the result as a new time series. This speeds up dashboard queries (Grafana or Prometheus’ built-in UI) because the computed result is already available in the TSDB instead of being computed on demand. Key links:

Prometheus: https://prometheus.io/
PromQL basics: https://prometheus.io/docs/prometheus/latest/querying/basics/
Grafana: https://grafana.com/

The image explains how Prometheus uses recording rules to periodically evaluate PromQL expressions and store the resulting time series, enhancing dashboard speed and providing aggregated results for use elsewhere.

How it works at a glance:

You define recording rules in one or more rule files (YAML).
Prometheus evaluates the rule expressions at a configured interval and writes the results to the TSDB as new metrics.
Dashboards and queries can then read those recorded metrics by name instead of recalculating expensive expressions.

Configure rule file loading

In prometheus.yml you point Prometheus to one or more rule files using the rule_files section. Globs are supported, so you can load all files from a directory. Example minimal configuration (note rule_files must list the file(s) or glob patterns to load):

global:
  scrape_interval: 5s
  evaluation_interval: 5s

rule_files:
  - /etc/prometheus/rules/*.yml

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ["localhost:9090"]

If you change rule files, Prometheus needs to reload its configuration. You can either restart the service or trigger a reload using the HTTP endpoint (POST /-/reload) or SIGHUP. Reloading avoids a full restart.

Rule file structure

A rule file contains one or more rule groups under the groups: key. Each group can define an interval (overrides global evaluation interval for that group) and a list of rules. Each rule has:

record: name for the resulting time series
expr: the PromQL expression to evaluate
labels (optional): to add/remove labels on the recorded series

Example skeleton:

groups:
  - name: <group name 1>
    interval: <evaluation interval>   # optional; overrides global evaluation_interval
    rules:
      - record: <rule_name_1>
        expr: <promql_expression_1>
        labels:
          <label_name>: <label_value>
      - record: <rule_name_2>
        expr: <promql_expression_2>
  - name: <group name 2>
    rules:
      # ...

Table: rule file keys

Field	Description	Example
`groups[].name`	Logical name grouping related rules	`node`
`groups[].interval`	Optional evaluation interval for the group	`15s`
`groups[].rules[].record`	Metric name to create in TSDB	`node_filesystem_free_percent`
`groups[].rules[].expr`	PromQL expression to evaluate	`100 * node_filesystem_free_bytes / node_filesystem_size_bytes`
`groups[].rules[].labels`	Optional labels to add or override	`job: "node"`

Parallel/Sequential evaluation model

Each rule group is evaluated in parallel with other groups.
Within a group, rules are evaluated sequentially in the order listed. This allows later rules to reference earlier recorded rules in the same group.

The image is a diagram showing rules organized into two groups, "Group1" and "Group2," with both sequential and parallel execution paths leading to a database icon.

Because rules inside a group run sequentially, any rule that references another rule must come after the rule it references. Otherwise the referenced recorded series may not exist yet.

Useful example expressions

Calculate percent memory free on a node:

100 - (100 * node_memory_MemFree_bytes / node_memory_MemTotal_bytes)

Calculate filesystem free percentage:

100 * node_filesystem_free_bytes / node_filesystem_size_bytes

Example recording rules

Create /etc/prometheus/rules.yml (or any file you reference from rule_files) with a group that runs every 15s and records the two metrics above:

groups:
  - name: example1
    interval: 15s
    rules:
      - record: node_memory_memFree_percent
        expr: 100 - (100 * node_memory_MemFree_bytes / node_memory_MemTotal_bytes)
      - record: node_filesystem_free_percent
        expr: 100 * node_filesystem_free_bytes{job="node"} / node_filesystem_size_bytes{job="node"}

After reloading Prometheus, you can query the recorded metric by name rather than the original expression: Example query result:

$ node_memory_memFree_percent
node_memory_memFree_percent{instance="192.168.1.168:9100", job="node"} 25

And for filesystem percentages:

$ node_filesystem_free_percent
node_filesystem_free_percent{device="/dev/sda2", instance="node1", job="node", mountpoint="/boot/efi"} 20
node_filesystem_free_percent{device="/dev/sda3", instance="node1", job="node", mountpoint="/"} 72
...

Chaining rules (reference recorded metrics)

Because rules in a group run sequentially, you can reference a recorded metric in a subsequent rule. Example:

groups:
  - name: example1
    interval: 15s
    rules:
      - record: node_filesystem_free_percent
        expr: 100 * node_filesystem_free_bytes{job="node"} / node_filesystem_size_bytes{job="node"}
      - record: node_filesystem_free_percent_avg
        expr: avg by(instance)(node_filesystem_free_percent)

Here, node_filesystem_free_percent_avg aggregates the recorded node_filesystem_free_percent metric. Because it appears after the recorded metric in the same group, the reference will resolve correctly.

Naming best practices for recording rules

Use a structured naming scheme composed of three parts separated by colons (or another consistent delimiter):

level — aggregation level (labels retained), typically includes job plus any target labels
metric — the base metric name
operation — aggregators/functions applied (e.g., rate_5m, sum, avg)

Example: job:method:path:http_errors:rate_5m (or a compact variant like job_method_path_http_errors_rate_5m depending on your conventions).

The image shows a diagram explaining record rule naming, specifically describing the components: level, metric_name, and operations, with definitions for each. It mentions how these elements relate to aggregation levels, metric names, and applied functions or aggregators.

Example: an HTTP errors counter instrumented with labels method and path and you want a 5m rate aggregated by job, method, path:

Aggregation level: job,method,path
Metric name: http_errors_total
Operation: rate_5m

Resulting recorded name (example convention): job_method_path_http_errors_rate_5m If you remove path from aggregation, change the level accordingly (e.g., job,method). Best practice: group all rules for a particular scrape job into a single rule group. That keeps related metrics together and ensures consistent evaluation intervals. Example grouping by job:

groups:
  - name: node            # all rules for job="node"
    interval: 15s
    rules:
      - record: node_memory_memFree_percent
        expr: 100 - (100 * node_memory_MemFree_bytes{job="node"} / node_memory_MemTotal_bytes{job="node"})
  - name: docker          # all rules for job="docker"
    interval: 15s
    rules:
      # docker-specific recording rules here

Add rules to Prometheus and reload

Create or edit rule file (example):

sudo vi /etc/prometheus/rules.yml
# paste your groups/rules (see examples above)

Ensure prometheus.yml contains the rule file path (or glob) under rule_files:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - "rules.yml"

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]
  - job_name: "node"
    static_configs:
      - targets: ["192.168.1.168:9100", "192.168.1.168:9200"]
  - job_name: "api"
    static_configs:
      - targets: ["192.168.1.168:8000"]
  - job_name: "ec2"
    ec2_sd_configs:
      - region: us-east-1

Reload Prometheus configuration:

Reload via HTTP API:

curl -X POST http://localhost:9090/-/reload

Or restart service:
sudo systemctl restart prometheus

Verify rules

Open the Prometheus UI:

Go to Status → Rules
Confirm rules are OK and evaluate at the expected interval
You can also query the recorded metric names in the expression browser to see current values.

Example rule display and query results:

record: node_filesystem_free_percent
expr:   100 * node_filesystem_free_bytes / node_filesystem_size_bytes{job="node"}

record: node_filesystem_free_percent_avg
expr:   avg by(instance) (node_filesystem_free_percent)

Example sample output:

node_filesystem_free_percent{device="/dev/sda3", fstype="ext4", instance="192.168.1.168:9100", job="node", mountpoint="/"} 16.005607045046514
...

This completes a concise guide to creating, naming, and using recording rules to improve query performance and enable efficient reuse of computed metrics in dashboards and alerts.

Introduction

Observability Fundamentals

Prometheus Fundamentals

PromQL

Dashboarding & Visualization

Application Instrumentation

Service Discovery

Push Gateway

Alerting

Monitoring Kubernetes

Conclusion

Scaling & Long Term Storage

Configure rule file loading

Rule file structure

Parallel/Sequential evaluation model

Useful example expressions

Example recording rules

Chaining rules (reference recorded metrics)

Naming best practices for recording rules

Add rules to Prometheus and reload

Verify rules

Watch Video

Practice Lab

​Configure rule file loading

​Rule file structure

​Parallel/Sequential evaluation model

​Useful example expressions

​Example recording rules

​Chaining rules (reference recorded metrics)

​Naming best practices for recording rules

​Add rules to Prometheus and reload

​Verify rules

Watch Video

Practice Lab

Configure rule file loading

Rule file structure

Parallel/Sequential evaluation model

Useful example expressions

Example recording rules

Chaining rules (reference recorded metrics)

Naming best practices for recording rules

Add rules to Prometheus and reload

Verify rules