- What aggregation operators do
- How to group results using
by(...) - How to exclude labels using
without(...) - Practical examples with
http_requestsandnode_cpu_seconds_total
Aggregation operators overview
Aggregation operators operate across all series in an instant vector by default. The most common are:Aggregation reduces multiple time series into fewer series. If you need to retain distinctions (e.g., per path or per instance), use
by(...) to group by label(s) before aggregation.Basic examples: sum, max, avg
Given the followinghttp_requests instant vector:
- Sum across all series:
- Max across all series:
- Average across all series:
Grouping with by(...)
Use by(<label1>, <label2>, ...) to keep the specified labels and aggregate across all others. This is how you “group by” a label.
Example: sum by path
path label and sums all series that share the same path.
Example: sum by method
by(...) to aggregate by combinations (e.g., by(instance, method)).
Excluding labels with without(...)
without(...) is the inverse of by(...). It aggregates across all labels except the ones listed. Use it when you want to drop specific labels and collapse across the rest.
Example:
instance and method labels, you can do:
sum by(instance, method) (http_requests).
Be careful: aggregation removes labels not listed in
by(...) (or not excluded by without(...)). If those labels are important for identifying series, include them in by(...).Real-world: node_cpu_seconds_total examples
Considernode_cpu_seconds_total, which typically has labels like cpu, instance, mode, and job. If you run:
instance:
instance, aggregating across cpu and mode.
If you want per-instance and per-CPU totals, group by both:
- instance A, cpu 0
- instance A, cpu 1
- instance B, cpu 0
- instance B, cpu 1
cpu and mode and aggregate everything else, use without(...). For example, to aggregate into per-instance totals via without:
cpu and mode from the result, leaving the aggregation keyed by the remaining labels (e.g., instance and job).
Quick tips
- Use
by(...)to preserve the labels you care about. - Use
without(...)to drop specific labels and aggregate across the rest. - Aggregation dramatically changes the dimensionality of your data — including or excluding labels will affect alerting and dashboarding logic.
- When building dashboards, prefer aggregations that match the level of granularity you want to visualize (per-instance, per-pod, per-cluster, etc.).