Skip to main content
In this lesson we cover how to work with Prometheus histogram metrics in PromQL: what each submetric represents, common queries (rates, averages, percentages), how to estimate quantiles and check SLOs, and trade-offs when designing buckets versus using summaries. A Prometheus histogram (for example, request_latency_seconds) exposes three kinds of time series: Here is a representative set of timeseries for a histogram:
  • _count (here 100) is the total number of requests.
  • _sum is the sum of observed latencies (seconds).
  • Each request_latency_seconds_bucket{le="X"} is the cumulative count of observations with value ≤ X. For example le="0.05" = 50 means 50 requests observed latency ≤ 0.05s.
Note that bucket counts are cumulative: the value at a larger le includes the counts of all smaller le buckets plus any observations between them. The le="+Inf" bucket should equal the _count series — if it does not, it usually indicates incorrect instrumentation or a bug.
Buckets are cumulative — when interpreting buckets, subtract the previous bucket if you need the number of observations that fall strictly between two boundaries.

​
Common queries

Use rate() or increase() on histogram submetrics (they are counters) depending on whether you want a rate or a total increase over a range.
  • Instantaneous per-second request rate (1m window):
  • Per-bucket rates (graphing bucket series over time):
  • Average request latency (over last 5m):
  • Fraction of requests ≤ a specific bucket (le="0.06"), using vector matching to ignore the le label on the right-hand side:
  • Number of observations between two bucket boundaries (5m window):
The image explains quantiles, stating they determine how many values in a distribution are above or below a certain limit and represent percentiles, with an example of the 90% quantile.

​
Quantiles from histograms

Prometheus provides histogram_quantile() to estimate quantiles from histogram buckets. The function takes a quantile (0..1) and a bucket series:
When aggregating across multiple instances, aggregate bucket counters by le first (so the quantile function sees a complete set of bucket boundaries), for example:
Example output (illustrative):
This indicates that ~75% of requests had latency ≤ 0.076s (estimated).
The image contains a text explanation about the histogram_quantile() function, discussing its approximation of quantile values and emphasizing the need for specific bucket settings to accurately measure Service Level Objectives (SLOs).
Important: histogram_quantile() uses linear interpolation between bucket boundaries — it produces an estimate, not an exact value. For SLO checks, ensure buckets are placed so interpolation around the SLO threshold is meaningful. For example, if your SLO requires 95% of requests ≤ 0.5s, include a bucket at le="0.5" and compute:
If the result is greater than 0.5 the SLO is missed. Keep in mind the reported quantile may be substantially influenced by bucket spacing around the threshold.
The image explains the limitations of quantile reporting using buckets, illustrating how a reported value of 0.4 could fall between bucket values of 0.2 and 0.5. It suggests adding more buckets for increased accuracy.
Adding more buckets improves interpolation accuracy where you need precision, but each bucket increases cardinality and Prometheus resource usage (memory, disk, and ingest cost). Choose bucket boundaries sparingly and focused around thresholds important for SLOs.
The image shows the Prometheus web interface on a Firefox browser, with a dropdown list of metric options related to request latency and totals.

​
Concrete histogram example

Corrected example for a /cars path (label order shown as method="GET", path="/cars"):
  • le="0.02" = 3 → 3 requests ≤ 0.02s.
  • le="0.05" = 5 → 5 requests ≤ 0.05s (includes those ≤ 0.02s).
Compute the 95th percentile from these buckets:
Example result (illustrative):
This estimate implies 95% of requests had latency ≤ 0.96s given the bucket placement and interpolation.

​
Summaries vs Histograms

A Prometheus summary (client-side summary) exposes quantiles directly rather than buckets. A summary typically provides:
  • request_latency_seconds{quantile="0.01"} 0.004
  • request_latency_seconds{quantile="0.5"} 0.0296
  • request_latency_seconds{quantile="0.95"} 0.06
  • request_latency_seconds_count — total samples
  • request_latency_seconds_sum — sum of observed values
Example:
Summaries return client-computed quantiles (no server interpolation), but you must decide which quantiles to record at instrumentation time and aggregation across instances is limited.
The image contrasts histograms and summaries, highlighting differences in bucket selection, client library impact, quantile selection, and server-side calculations.

​
Design trade-offs

Keep these trade-offs in mind when designing metrics and SLO checks.
If your SLO depends on a specific latency threshold, add a bucket exactly at that threshold (or have buckets tightly spaced around it). Otherwise histogram_quantile() can only estimate and might hide how close you are to the SLO boundary.

​
References and further reading

Keep bucket cardinality focused on the thresholds and percentiles that matter for your SLOs to balance accuracy and resource usage.

Watch Video

Practice Lab