- Track latency distributions, not just averages.
- Compute quantiles using Prometheus (
histogram_quantile()). - Aggregate latency across many instances with histogram buckets.
- Flask: https://flask.palletsprojects.com/en/latest/
- Prometheus Python client: https://prometheus-client.readthedocs.io/en/latest/
- histogram_quantile(): https://prometheus.io/docs/prometheus/latest/querying/functions/#histogram_quantile
Approach (step-by-step)
- Create a histogram metric with the label names you need (for example,
pathandmethod). - Record a timestamp when the request starts (use
@app.before_request). - Record another timestamp right before sending the response (use
@app.after_request), subtract to get latency, then call.observe()on the histogram.
@app.before_requestruns before each request handler and is the ideal place to capture the start timestamp.@app.after_requestruns when a view returns but before the response is sent. Use it to compute and record latency. Always return the response fromafter_request.
- Keep label ordering consistent. The histogram is created with
labelnames=['path', 'method']. When using positional labels, pass values in the same order:.labels(request.path, request.method). You can also use keyword labels:.labels(path=request.path, method=request.method). - Record metric values by calling
.observe(value), wherevalueis the measured latency in seconds (a float). - Expose metrics via the Prometheus client (here
start_http_server(8000)), and configure Prometheus to scrape that endpoint.
/metrics the histogram exposes bucket counts, the total count, and the sum. Example scraped output:
Customizing buckets
Default buckets are useful for many web apps, but you may need different boundaries for your latency distribution. Supply abuckets= list when creating the Histogram:
Summary vs Histogram: when to use which
- Histograms
- Allow you to configure buckets and compute quantiles across instances using Prometheus functions like
histogram_quantile(). - Are usually the best choice when you need aggregated, cluster-wide quantiles.
- Allow you to configure buckets and compute quantiles across instances using Prometheus functions like
- Summaries
- Can provide client-side quantiles (e.g., 0.5, 0.9, 0.99) without bucket boundaries.
- Are limited because client-side quantiles do not aggregate well across instances.
Example: switching to a Summary is as simple as replacing
Histogram with Summary:
Histograms and summaries serve different use cases. Histograms are preferred when you need configurable buckets and aggregated quantiles via Prometheus’s
histogram_quantile() function. Summaries provide client-side quantiles per instance but do not aggregate quantiles across multiple instances effectively.The Prometheus Python client has fewer features for Summary (for example, client-side quantile objectives) compared with some other language clients. If you require aggregated quantiles across multiple instances, prefer Histograms and calculate quantiles in Prometheus with
histogram_quantile().- Prometheus Statistics and Histograms: https://prometheus.io/docs/concepts/metric_types/
- Prometheus query examples: https://prometheus.io/docs/prometheus/latest/querying/basics/
- Flask documentation: https://flask.palletsprojects.com/en/latest/