Aggregation

In this lesson we cover PromQL aggregation operators — functions that combine the elements of an instant vector to produce a new instant vector (often with fewer series). Aggregations are useful for producing totals, averages, min/max values, and group-based rollups across label dimensions.

A presentation slide titled "Aggregation operators" showing a brief explanation of aggregating instant vectors and a table listing aggregator functions (Sum, Min, Max, Avg, Group, Stddev, etc.) with their descriptions. The words "aggregate" and "fewer" are highlighted in the header text.

Aggregation operators reduce an instant vector into coarser results. Use by(...) to keep specific labels in the output, or without(...) to drop labels from the aggregation key.

Common aggregation functions

Function	Purpose	Example
`sum()`	Sum of all matching series values	`sum(http_requests)`
`min()`	Minimum value among series	`min(cpu_usage)`
`max()`	Maximum value among series	`max(cpu_usage)`
`avg()`	Average of series values	`avg(request_duration_seconds)`
`stddev()` / `stdvar()`	Standard deviation / variance	`stddev(metric)`
`count()`	Count of series	`count(up)`
`count_values("label", metric)`	Count of series by value	`count_values("mode", node_cpu_seconds_total)`
`group()`	Group samples into vectors (rarely used)	`group(metric)`

Simple aggregation examples

Given an http_requests instant vector (each line is a distinct series sample):

http_requests{method="get",  path="/auth"}  3
http_requests{method="post", path="/auth"}  1
http_requests{method="get",  path="/user"}  4
http_requests{method="post", path="/user"}  8
http_requests{method="post", path="/upload"} 2
http_requests{method="get",  path="/tasks"} 4
http_requests{method="put",  path="/tasks"} 6
http_requests{method="post", path="/tasks"} 1
http_requests{method="get",  path="/admin"} 3
http_requests{method="post", path="/admin"} 9

A plain aggregation over the whole set:

sum(http_requests)
# -> {} 41   # 3+1+4+8+2+4+6+1+3+9 = 41

Other straightforward reductions:

max(http_requests)
avg(http_requests)
# -> {} 4.1

Grouping results with `by(...)`

To aggregate while preserving one or more label dimensions, use by(...). Series that share the listed labels are combined. Example: group by path to sum requests per path:

sum by(path) (http_requests)
# -> {path="/auth"}   4   # 3+1
# -> {path="/user"}  12   # 4+8
# -> {path="/upload"} 2   # 2
# -> {path="/tasks"} 11   # 4+6+1
# -> {path="/admin"} 12   # 3+9

Group by method to get totals per HTTP method:

sum by(method) (http_requests)
# -> {method="get"}  14  # 3+4+4+3
# -> {method="post"} 21  # 1+8+2+1+9
# -> {method="put"}   6  # 6

Multi-label grouping

If time series include an instance label (two nodes: node1 and node2), the dataset might look like:

http_requests{method="get",  path="/auth", instance="node1"}  3
http_requests{method="post", path="/auth", instance="node1"}  1
http_requests{method="get",  path="/user", instance="node1"}  4
http_requests{method="post", path="/user", instance="node1"}  8
http_requests{method="post", path="/upload", instance="node1"} 2
http_requests{method="get",  path="/tasks", instance="node1"} 4
http_requests{method="put",  path="/tasks", instance="node1"} 6
http_requests{method="post", path="/tasks", instance="node1"} 1
http_requests{method="get",  path="/admin", instance="node1"} 3
http_requests{method="post", path="/admin", instance="node1"} 9

http_requests{method="get",  path="/auth", instance="node2"} 13
http_requests{method="post", path="/auth", instance="node2"} 11
http_requests{method="get",  path="/user", instance="node2"} 14
http_requests{method="post", path="/user", instance="node2"} 18
http_requests{method="post", path="/upload", instance="node2"} 12
http_requests{method="get",  path="/tasks", instance="node2"} 14
http_requests{method="put",  path="/tasks", instance="node2"} 16
http_requests{method="post", path="/tasks", instance="node2"} 11
http_requests{method="get",  path="/admin", instance="node2"} 13
http_requests{method="post", path="/admin", instance="node2"} 19

You can aggregate by both instance and method:

sum by(instance, method) (http_requests)
# -> {instance="node1", method="get"}  14  # 3+4+4+3
# -> {instance="node1", method="post"} 21  # 1+8+2+1+9
# -> {instance="node1", method="put"}   6  # 6
# -> {instance="node2", method="get"}  54  # 13+14+14+13
# -> {instance="node2", method="post"} 71  # 11+18+12+11+19
# -> {instance="node2", method="put"}  16  # 16

The `without(...)` clause

without(...) is the inverse of by(...) — it tells Prometheus which labels to exclude from the aggregation key. The result keeps all other labels. Using the dataset above, sum without(path) (http_requests) removes the path label and aggregates across that dimension. For this dataset it is equivalent to sum by(instance, method) (http_requests):

sum without(path) (http_requests)
# -> {instance="node1", method="get"}  14
# -> {instance="node1", method="post"} 21
# -> {instance="node1", method="put"}   6
# -> {instance="node2", method="get"}  54
# -> {instance="node2", method="post"} 71
# -> {instance="node2", method="put"}  16

Be cautious when grouping by high-cardinality labels (for example, pod, instance, or user IDs): by(...) can produce many result series and increase memory/CPU usage. Prefer grouping by stable, low-cardinality labels for dashboards and alerts.

Real-world example: CPU seconds

Consider node_cpu_seconds_total, where each series has labels like instance, cpu, mode, and job:

node_cpu_seconds_total{cpu="0", instance="192.168.1.168:9100", job="node", mode="idle"}    13084.27
node_cpu_seconds_total{cpu="0", instance="192.168.1.168:9100", job="node", mode="iowait"}  2.65
node_cpu_seconds_total{cpu="0", instance="192.168.1.168:9100", job="node", mode="user"}    31.89
...
node_cpu_seconds_total{cpu="1", instance="192.168.1.168:9200", job="node", mode="user"}    31.97

A plain sum(node_cpu_seconds_total) aggregates every CPU and instance into a single number. More useful rollups:

Total CPU seconds per instance:

sum by(instance) (node_cpu_seconds_total)
# -> {instance="192.168.1.168:9100"} 26344.75
# -> {instance="192.168.1.168:9200"} 26363.22

Totals per CPU per instance:

sum by(instance, cpu) (node_cpu_seconds_total)
# -> {cpu="0", instance="192.168.1.168:9100"} 13213.80
# -> {cpu="0", instance="192.168.1.168:9200"} 13208.07
# -> {cpu="1", instance="192.168.1.168:9100"} 13190.79
# -> {cpu="1", instance="192.168.1.168:9200"} 13185.06

Keep instance and job but drop cpu and mode:

sum without(cpu, mode) (node_cpu_seconds_total)
# -> {instance="192.168.1.168:9100", job="node"} 26344.75
# -> {instance="192.168.1.168:9200", job="node"} 26363.22

Quick reference

Intent	Syntax
Aggregate all series into a single value	`sum(metric)`
Aggregate and keep `label1`	`sum by(label1) (metric)`
Aggregate and keep `label1` and `label2`	`sum by(label1, label2) (metric)`
Aggregate and drop `labelA` and `labelB`	`sum without(labelA, labelB) (metric)`

Links and references

Prometheus aggregation docs: https://prometheus.io/docs/prometheus/latest/querying/operators/#aggregation-operators
PromQL basics: https://prometheus.io/docs/prometheus/latest/querying/basics/

Summary: Aggregation operators let you reduce instant vectors into summarized results. Use by(...) to declare which labels to preserve in the result, and without(...) to remove labels from the aggregation key — both are essential for meaningful, efficient rollups in Prometheus queries.

Introduction

Observability Fundamentals

Prometheus Fundamentals

PromQL

Dashboarding & Visualization

Application Instrumentation

Service Discovery

Push Gateway

Alerting

Monitoring Kubernetes

Conclusion

Scaling & Long Term Storage

Common aggregation functions

Simple aggregation examples

Grouping results with `by(...)`

Multi-label grouping

The `without(...)` clause

Real-world example: CPU seconds

Quick reference

Links and references

Watch Video

Practice Lab

​Common aggregation functions

​Simple aggregation examples

​Grouping results with by(...)

​Multi-label grouping

​The without(...) clause

​Real-world example: CPU seconds

​Quick reference

​Links and references

Watch Video

Practice Lab

Common aggregation functions

Simple aggregation examples

Grouping results with `by(...)`

Multi-label grouping

The `without(...)` clause

Real-world example: CPU seconds

Quick reference

Links and references