Recap Structural Prop of Metrics

​

Monotonicity

• Definition: Whether a metric only increases (monotonic) or can both increase and decrease (non‑monotonic).
• Typical types:
  • Monotonic: counters (e.g., http_requests_total, errors_total) — these only increase until a reset.
  • Non‑monotonic: gauges (e.g., CPU usage, temperature, current queue length) — values can rise and fall.
• Why it matters:
  • Monotonic counters are ideal for computing rates and totals. You must detect resets (e.g., process restart) when computing deltas.
  • For non‑monotonic gauges, use last‑value or instantaneous aggregation semantics.
• Quick rule: apply rate computations only to monotonic counters.

rate(http_requests_total[5m])

​

Temporality

• Definition: How values are reported over time — whether they represent totals, interval deltas, or instantaneous readings.
• Modes and when to use them:
  • Cumulative: running total since some epoch (use for overall totals, e.g., lifetime requests).
  • Delta: reported change during each collection interval (use for “new” events per interval).
  • Gauge (instantaneous): value at the collection time (use for current state like temperature or memory usage).
• Practical guidance:
  • Use cumulative when you want to compute rates or long‑term totals.
  • Use delta when the instrumentation already emits interval deltas and you want to avoid re‑deriving them.
  • Use gauge when the point‑in‑time value is the meaningful metric.

​

Aggregation

• Definition: How raw measurements are summarized for storage and visualization.
• Common aggregations:
  • sum — totals across groups (useful for load).
  • count — number of samples or occurrences.
  • last-value — current gauge value.
  • histogram / percentiles — latency or size distributions.
• Tradeoffs:
  • Histograms capture distribution and percentiles but cost more storage and processing.
  • Simple sums/counts are lightweight but lose distributional detail.
• Choose the aggregation that preserves the information you need for SLOs and alerts while controlling storage cost.

​

Dimensions (labels / attributes)

• Definition: Key–value attributes attached to each metric sample (for example: method=GET, route=/login).
• Uses:
  • Filtering, grouping, and drilling down to subsets of traffic (e.g., requests for GET /login).
  • Enabling targeted alerts and dashboards.
• Cardinality considerations:
  • Cardinality = number of unique label combinations (time series). Each new high‑cardinality label multiplies series count.
  • High‑cardinality labels (e.g., user_id, session_id) can drastically increase storage and query costs and impair performance.
• Strategies to control cardinality:
  • Avoid raw unique identifiers as labels.
  • Use sampling, hashing, bucketing, or roll‑ups (e.g., user_bucket=0..100) to reduce uniqueness.
  • Precompute and emit lower‑cardinality dimensions when possible.

​

Quick reference table

​

Practical tips

• When building counters, always handle resets: compute delta = current - previous and, if current < previous, treat as a reset.
• Use histograms to capture latency distributions without storing every raw measurement.
• Monitor and set alerts for series explosion (sudden cardinality growth) to catch instrumentation issues early.

​

Links and further reading

• OpenTelemetry Metrics
• Prometheus Documentation — Counters and Gauges
• Best Practices for Metric Cardinality