Skip to main content
In this lesson we’ll cover best practices for naming and instrumenting metrics so they are predictable, unambiguous, and easy to consume with Prometheus and other monitoring tools.

​
Metric naming conventions

  • Use snake_case for metric names: lowercase words separated by underscores (for example http_request_total).
  • Prefix metric names with the application or library name to indicate ownership (for example postgres_ for Postgres-related metrics).
  • Always include the unit in the metric name (use base units such as seconds, bytes, meters) rather than prefixed units (milliseconds, kilobytes).
  • Rely on client libraries to produce type-specific suffixes. Counters should end with _total; do not manually add _sum, _bucket, or _count for histograms/summaries — client libraries export those.
The image provides guidelines for naming metrics, specifying the use of snake_case, and emphasizing that metrics should start with the application/library name.
For example, metrics related to Postgres should begin with postgres_, e.g. postgres_connection_errors. After the prefix, use descriptive words (e.g., queue_size) and append the unit where applicable (e.g., _seconds, _bytes). Units avoid ambiguity about scale (is latency in seconds or milliseconds?). Prometheus client libraries and conventions prefer base units (seconds, bytes).
The image provides guidelines on naming metrics, emphasizing the inclusion of units like seconds and bytes, using a structure like library_name_unit_suffix.
Counters must end with _total. Do not manually add _sum, _bucket, or _count — those are created by client libraries for histogram/summary metric types.
Common recommended structure:
  • <library>_<metric_name>_<unit>_<optional_suffix>
Good examples:
Bad examples and why they are problematic:
The image shows a comparison of proper and incorrect metric names, with examples such as "process_cpu_seconds" and "http_requests_total" being proper, and "container_docker_restarts" and "http_requests_sum" being incorrect.
The image lists naming conventions for metrics, categorizing them into "Proper Metric Names" and "Incorrect Metric Names" with examples for each category.

​
Common suffixes and metric types

For more details on conventions, see the Prometheus best practices and instrumentation guides:

​
What to instrument

Decide which metrics to collect based on the nature of your application. The three common categories and recommended metrics are:

​
1) Online serving systems

Online systems expect immediate responses (databases, web servers, APIs). Instrument the following:
  • Total number of requests/queries (*_total)
  • Error counts (*_errors_total)
  • Request latency (*_seconds, often as histograms or summaries)
  • Number of in-progress requests (gauge)
The image is a slide about online-serving systems, highlighting the expectation of immediate responses and including metrics like the number of queries, errors, latency, and ongoing requests.
Suggested metrics for an HTTP server:

​
2) Offline processing services

These are long-running background systems with multiple stages. Instrument per-stage and pipeline-level metrics:
  • Amount of work processed (counter)
  • Queue/backlog size (gauge)
  • Work in progress (gauge)
  • Processing rate (derived from counters)
  • Errors per stage
The image explains offline processing services, noting they're not actively monitored, and stages involve measuring queued work, work in progress, and processing rate.
Example per-stage metrics:

​
3) Batch jobs

Batch jobs run on a schedule and are often short-lived. Because Prometheus may not scrape short-lived processes, use a Pushgateway for job-level metrics that must be retained until scraped. Instrument the following for batch jobs:
  • Per-stage processing time (_seconds)
  • Overall job runtime (_seconds)
  • Success and failure counts (myjob_success_total, myjob_failure_total)
  • Number of items processed (*_processed_total)
  • Backlog/queued items before start
The image describes the differences between batch jobs and offline-serving systems, noting that batch jobs run on a regular schedule while offline systems run continuously, and mentions the need for a pushGateway for non-continuous tasks.
When using a Pushgateway, label pushed metrics clearly (e.g., job and instance identifier). Do not use the Pushgateway for ephemeral per-request metrics — it’s intended for short-lived batch jobs and service-level metrics.

​
Practical tips

  • Keep metric names consistent across services to simplify querying and alerting.
  • Use labels for dimensions (e.g., method, status_code, region) instead of encoding dimensions into metric names.
  • Prefer histograms for latency distributions and counters for cumulative counts.
  • Document your metrics (what they measure, units, and labels) so consumers and SREs can use them correctly.

​
References

Watch Video

Practice Lab