Skip to main content
In this lesson we cover how Prometheus exposes histogram and summary metrics, how to read them, and practical PromQL examples for common tasks: rates, averages, bucket percentages, and quantiles. This guide preserves example telemetry dumps and shows how to extract meaningful SLOs and distributions from them.

​
What a histogram exposes

A histogram metric in Prometheus is represented by three related time series:
  • The _count series — the total number of observations.
  • The _sum series — the sum of all observed values.
  • The _bucket series — cumulative bucket counts with an le label (less-or-equal).
Example: a request latency histogram for the /articles path:
Reading histogram buckets:
  • Each request_latency_seconds_bucket{le="X"} value is the cumulative count of observations with value <= X.
  • Buckets are cumulative: the count at le="0.03" includes counts from le="0.01" and le="0.02", plus any observations between 0.02 and 0.03.
  • The le="+Inf" bucket contains all observations (equal to _count).
For example:
  • request_latency_seconds_bucket{le="0.05", path="/articles"} 50 means 50 requests took <= 0.05 seconds.
  • request_latency_seconds_bucket{le="0.04", path="/articles"} 44 means 44 requests took <= 0.04 seconds.
When working with counters like _count and _sum, use rate() (or irate() for instant rates) over an appropriate lookback window. This converts counters into per-second rates suitable for further math and charting.

​
Common PromQL patterns with histograms

  • Rate of incoming requests per second (1m window):
  • Rate for every bucket (to visualize distribution over time):
  • Total observed value (e.g., sum of latencies or bytes):
  • Rate of increase for that sum:
  • Average latency over the past 5 minutes (sum rate divided by count rate):
  • Fraction (percentage) of requests that fell into a specific bucket le="0.06" over 5 minutes:
  • Number of observations that fell between two bucket boundaries (approximate):
Observations in (0.02, 0.05] ≈

​
Histogram quantiles

Quantiles (percentiles) tell you the value below which a given percentage of observations fall. Prometheus computes approximate quantiles from histogram buckets using histogram_quantile().
A presentation slide titled "Quantiles" defining them as percentiles that determine how many values in a distribution lie above or below a certain limit. It notes that the 90% quantile is the value below which 90% of the data falls.
Example: calculate the 95th percentile latency (uses the histogram buckets):
The returned value is an estimate (Prometheus does linear interpolation between bucket boundaries). Example result:
This indicates that approximately 95% of requests had latency <= 0.96s.

​
Accuracy and bucket placement

  • histogram_quantile() is an approximation derived by linear interpolation between bucket boundaries.
  • To improve accuracy around a specific SLO, include a bucket whose le equals the SLO threshold. For example, if your SLO is 95% of requests <= 0.5s, ensure you have a bucket with le="0.5".
  • If you require higher precision near an SLO boundary, add more buckets there — but do so sparingly because each bucket creates an additional time series.
Adding many buckets increases cardinality: more time series, higher memory and disk usage, and slower insert performance. Add buckets only where you need precision.
A presentation slide titled "Quantile" explaining that each histogram bucket is stored as a separate time series. It warns that too many buckets lead to high cardinality, high RAM usage, and high disk space.

​
Practical histogram example

Another histogram dump for the /cars endpoint:
From the example:
  • le="0.02" has a count of 3 → three requests <= 0.02s.
  • le="0.05" has a count of 10 → ten requests <= 0.05s (this includes smaller buckets).
Compute the 95th percentile:
This might return:
Meaning the estimated 95th percentile is ~0.96s.

​
Summary metrics (Summaries)

Summaries are an alternative to histograms. They expose quantiles directly (computed on the client) instead of storing bucketed counts. A summary typically contains:
  • The count (_count) — total number of observations.
  • The sum (_sum) — sum of all observed values.
  • Quantile timeseries — with a quantile="<q>" label showing the observed value at that quantile.
Example summary output:
Interpretation:
  • request_latency_seconds{quantile="0.7"} 0.0365 means the 70th percentile is 0.0365s — 70% of observations are <= 0.0365s.

​
Histogram vs Summary — key differences

Reference: Prometheus documentation for histograms and summaries — https://prometheus.io/docs/practices/histograms/
A presentation slide titled "Histogram vs Summary" comparing the two: the left column lists histogram points (pick bucket sizes, less client load, any quantile, server computes quantiles) and the right column lists summary points (quantiles must be predefined, more client work, limited quantiles, minimal server cost).

​
Takeaways

  • Use histograms when you need flexible, server-side quantile estimation and are willing to manage bucket boundaries.
  • Use summaries when the client can compute and expose the specific quantiles you need.
  • Always consider cardinality: each bucket or quantile creates additional time series. Add resolution only where it matters to your SLOs.
Further reading:

Watch Video

Practice Lab