Two samples match only if their label sets are identical (unless you explicitly change the matching behavior with
on(...), ignoring(...), group_left, or group_right).How matching works (high level)
When PromQL evaluates a binary operation between two instant vectors, it attempts to pair each left-hand series with one or more right-hand series based on their labels. By default, every label name and value must be identical on both sides for a pair to match. If labels differ or are missing, those series won’t be paired.Simple example: filesystem available / size
Given two metrics:node_filesystem_avail_bytes— available bytes per filesystemnode_filesystem_size_bytes— total size bytes per filesystem
instance, job, and mountpoint here). If any label differs or is present on only one side, the pair will not match.
Examples of non-matching situations:
- Different
instancevalue → no match. - Different
mountpointvalue → no match. - One series has an extra label (e.g.,
device) and the other does not → no match.
Matching by a subset of labels: on(...) vs ignoring(...)
Use on(...) or ignoring(...) to control which labels are considered during matching:
Both change label comparison rules;
on explicitly lists labels to match, while ignoring lists labels to omit from consideration.
Example scenario: error counts per HTTP method and code, and total requests per method.
Samples:
http_errors has an extra code label:
method:
code="500"):
Example: match by CPU only
If two vectors havecpu and mode labels but you want to combine them only by cpu, use either:
cpu, ignoring mode, producing per-CPU sums.
Matching cardinality: one-to-one vs many-to-one
Vector matching can be one-to-one (each element on the left matches exactly one on the right) or many-to-one (elements on one side match several elements on the other). PromQL requires you to be explicit for many-to-one matches usinggroup_left or group_right.
When multiple series on one side match a single series on the other side, you must explicitly use
group_left or group_right. Otherwise the query will error with a many-to-one ambiguity.path (different error codes) and right side is one per path:
- Use
group_leftwhen the left operand is the “many” side (keeps left-side extra labels). - Use
group_rightwhen the right operand is the “many” side.
error label (left is many):
group_right instead.
Full filesystem example (real labels)
A real-world query:device, fstype, instance, job, mountpoint). Example computed percentages:
error), use on(...), ignoring(...), and group_left/group_right to adjust matching behavior.
Summary
- By default, PromQL requires identical labels on both sides for vector matching.
- Use
on(...)orignoring(...)to control which labels are considered during matching. - When one side would match multiple series on the other side, use
group_leftorgroup_rightto make many-to-one relationships explicit. - Always verify the resulting label set after matching — extra labels are preserved only when using the appropriate grouping modifier.