Skip to main content
In this lesson we cover vector matching in PromQL—how Prometheus pairs elements from two instant vectors when performing element-wise arithmetic and how to control that matching using label selectors and grouping modifiers.

​
Element-wise operations between vectors

PromQL can operate on two instant vectors element-wise. For example, to compute the percentage of free space per filesystem, divide available bytes by total size and multiply by 100:
Query:
This divides each left-hand sample by the corresponding right-hand sample and multiplies by 100 to yield a percentage.

​
Label-matching rule (exact by default)

By default, PromQL only pairs two samples when their complete set of labels and values are identical. The matching process checks every label on the left sample and tries to find a sample on the right with the exact same label names and values.
All labels on the two vectors must match exactly unless you use on(...) or ignoring(...) to control which labels are used for matching.
If labels differ (or one sample has an extra label), they will not match:
Because instance differs, these samples will not pair.

​
Controlling which labels are used for matching

Use on(...) to list the label names to match on, or ignoring(...) to exclude certain labels from the matching. This is vital when one metric has additional label dimensions that should not prevent matching. Example: compute the percentage of 500 errors per HTTP method when error metrics include a code label, but request metrics do not. Input metrics:
Options to match by method:
  • Ignore the code label on the errors side:
  • Or explicitly match on method:
Only methods present on both sides will appear. In this dataset, put (only in http_errors) and del (only in http_requests) will be dropped from the output. You can list multiple labels in on(...):

​
Example: match on CPU but ignore mode

When metrics include different operational modes but you only want to match by CPU: vector1:
vector2:
Use on(cpu) or ignoring(mode) to match samples by cpu only.
This image illustrates vector matching keywords using two vectors, showing how elements are matched and summed based on specified labels like "cpu" and "mode."

​
Matching modes: one-to-one, many-to-one, one-to-many

  • One-to-one (default): each element on the left matches at most one element on the right.
  • Many-to-one / one-to-many: when multiple samples on one side match a single sample on the other side, you must explicitly allow this with group_left() or group_right().
When PromQL detects multiple matches for a single sample without an explicit grouping modifier, it returns an error: “multiple matches for labels”. Example (many-to-one): error counts per path and error vs. total requests per path. Input:
A naive query:
returns an error due to multiple http_errors entries for each path. Tell PromQL that the left-hand side may have multiple entries matching a single right-hand entry with group_left():
group_left() preserves the extra labels from the left-hand side (for example error) in the output.
The image explains the Many-To-One matching concept in PromQL, where each element on the "one" side can match with multiple elements on the "many" side, using an example of HTTP errors and requests. It demonstrates how group_left is used to match elements from the right side with multiple elements from the left side based on the path.
Use group_right() when the right-hand side may have multiple matching samples per single left-hand sample (one-to-many in the opposite direction). The group_*() modifier may be used with or without a label list; providing a label list preserves those labels from the many side in the result.

​
Prometheus UI example: filesystem availability percentage

Each filesystem time series commonly includes labels like device, fstype, instance, job, and mountpoint. When the label sets align exactly between node_filesystem_avail_bytes and node_filesystem_size_bytes, dividing them yields the percentage available for that timeseries:
If labels match perfectly, this query will return the expected percentage per filesystem.
The image shows a Prometheus web interface displaying queries for filesystem metrics, specifically node_filesystem_avail_bytes and node_filesystem_size_bytes, with data in a tabular format.
If labels do not match (for example one metric has an extra error label), the division returns no results unless you control matching with on(...), ignoring(...), and—if multiple matches occur—group_left() or group_right().
The image shows a Prometheus web interface displaying query results for file system usage metrics, including the percentage of available bytes and specific device details. It includes tables with metric names, device instances, and mount points.

​
Example: HTTP errors vs. HTTP requests totals

When http_errors_total includes an error label and http_requests_total does not:
A direct division will yield nothing because label sets differ. Exclude error from matching:
If multiple http_errors_total samples (different error values) match a single http_requests_total sample, use group_left() to explicitly allow many-to-one matching and retain the error label in results:
The image shows a Prometheus monitoring dashboard displaying file system metrics and HTTP requests with data in a tabular format. It includes queries for file system usage and HTTP error totals.

​
Matching modes at a glance

Note: In the table, A and B represent vectors; any label selectors like {path="/cats"} should be written as PromQL expressions in queries.
  • Always inspect the label sets of the two vectors (label_join, label_replace, or the Prometheus UI) before combining them to avoid silent drops or errors.
  • Use on(...) to restrict matching to specific key labels, and ignoring(...) to exclude irrelevant labels.
  • When allowing multiple matches, choose group_left() or group_right() according to which side holds the extra label dimensions you want to preserve.
  • To explicitly keep only select labels from the many side, supply them to group_left(<labels...>) or group_right(<labels...>).
Further reading: Summary
  • PromQL defaults to strict one-to-one label matching.
  • Use on(...) / ignoring(...) to control which labels are considered for matching.
  • Use group_left() / group_right() to explicitly allow many-to-one or one-to-many matches and to preserve labels from the many side when needed.

Watch Video