Metrics

In this lesson we explain how Prometheus metrics are structured, how Prometheus stores them, and best practices for designing metric names and labels. A Prometheus metric has three main properties:

Name — a descriptive identifier for the metric.
Labels — zero or more key/value pairs that add dimensions to the metric.
Value — the metric’s numeric value for a particular timestamp (collected when Prometheus scrapes the target).

Example metric format and a single sample line:

<metric_name>{<label_1="value_1">,<label_N="value_N">} <metric_value>
node_cpu_seconds_total{cpu="0",mode="idle"} 258277.86

This line is the node_cpu_seconds_total metric (exported by node_exporter). It represents the cumulative number of seconds the CPU has spent in a specific mode. Labels tell us which CPU (cpu="0") and which mode (mode="idle") this sample refers to. If the machine has multiple CPUs, Prometheus will store one sample per CPU (each with a different cpu label). A set of example mode metrics for CPU 0:

node_cpu_seconds_total{cpu="0",mode="idle"} 258277.86
node_cpu_seconds_total{cpu="0",mode="iowait"} 61.16
node_cpu_seconds_total{cpu="0",mode="irq"} 0
node_cpu_seconds_total{cpu="0",mode="nice"} 61.12
node_cpu_seconds_total{cpu="0",mode="softirq"} 6.63
node_cpu_seconds_total{cpu="0",mode="steal"} 0
node_cpu_seconds_total{cpu="0",mode="system"} 372.46
node_cpu_seconds_total{cpu="0",mode="user"} 3270.86

When Prometheus scrapes a target it stores a timestamp for each sample. Timestamps are Unix epoch seconds:

1668215300

This value is the number of seconds since the Unix epoch (January 1, 1970 UTC). Dashboarding tools typically convert this to your local timezone.

Prometheus stores a sequence of timestamped samples for each unique metric + label set — this sequence is called a time series. A time series is uniquely identified by the metric name together with its labels.

Example of how multiple time series arise from identical metric names but different label combinations (two servers, each with two CPUs):

node_filesystem_files{device="sda2",instance="server1"}
node_filesystem_files{device="sda3",instance="server1"}
node_filesystem_files{device="sda2",instance="server2"}
node_filesystem_files{device="sda3",instance="server2"}

node_cpu_seconds_total{cpu="0",instance="server1"}
node_cpu_seconds_total{cpu="1",instance="server1"}
node_cpu_seconds_total{cpu="0",instance="server2"}
node_cpu_seconds_total{cpu="1",instance="server2"}

Even with only two metric names here, there are eight time series because each unique label combination defines a separate series. Prometheus metrics can include two optional metadata lines:

# HELP node_disk_discard_time_seconds_total Total number of seconds spent by all discards.
# TYPE node_disk_discard_time_seconds_total counter
node_disk_discard_time_seconds_total{device="sda"} 0
node_disk_discard_time_seconds_total{device="sr0"} 0

Metric types

Prometheus exposes four primary metric types. Use the type that best matches what you are measuring.

Type	Description	Typical use cases
Counter	Monotonically increasing value (only goes up, or resets on restart).	Totals such as requests served, exceptions, or completed jobs.
Gauge	A value that can go up and down.	Current states such as CPU utilization, free memory, or number of concurrent requests.
Histogram	Buckets observations and counts how many fall below each bucket boundary (cumulative).	Response times, request payload sizes — useful when you need aggregation across instances.
Summary	Calculates quantiles client-side (e.g., 50th, 90th, 99th percentiles).	Latency percentiles per instance where client-side quantile calculation is required.

Counter — a monotonically increasing value. Useful for cumulative totals.

A dark-themed presentation slide titled "Counter" with two bullet points explaining a counter metric and three rounded boxes labeled "Total # requests," "Total # Exceptions," and "Total # of job executions."

Gauge — represents a value that can both increase and decrease. Use it for instantaneous measurements (CPU usage, number of active connections, queue lengths).
Histogram — groups observations into configurable buckets and reports cumulative counts for each bucket. For example, with response-time buckets at 0.2s, 0.5s, and 1s, a histogram can tell you how many requests finished in <0.2s, <0.5s, and <1s. Bucket counts are cumulative, so the count for the 0.5s bucket includes the 0.2s bucket.

A dark-themed infographic titled "Histogram" that explains histograms and shows example buckets for "Response Time" and "Request size." It includes a small 3D bar chart of response-time categories (e.g., <0.3s, <0.5s, <1s, <2s) and corresponding bucket thresholds.

Summary — provides client-side computed quantiles (percentiles) and a count/sum of observations. Summaries produce per-instance quantiles and therefore cannot be aggregated across instances as easily as histograms.

When to choose histogram vs. summary

Use histograms when you need to aggregate latency distributions across multiple instances (e.g., cluster-wide 95th percentile).
Use summaries when you need accurate, per-instance percentiles and do not require cross-instance aggregation.

Metric and label naming rules

Metric names may contain ASCII letters, digits, underscores, and colons. They must match:

[a-zA-Z_:][a-zA-Z0-9_:]*

Note: colons are reserved for Prometheus recording rule names; avoid using them in custom metric names. Label names must match:

[a-zA-Z_][a-zA-Z0-9_]*

Labels are powerful and should be used to slice and group metrics by dimensions you will actually query. Avoid creating one metric per dimension value. Bad (high maintenance and high-cardinality) approach:

# Not ideal: one metric per endpoint
requests_auth_total
requests_user_total
requests_products_total
requests_cart_total
requests_orders_total

Better approach — single metric with a path label:

# Preferred: single metric with a label for the endpoint
requests_total{path="/auth"}
requests_total{path="/user"}
requests_total{path="/products"}
requests_total{path="/cart"}
requests_total{path="/orders"}

Then you can aggregate across all paths:

sum(requests_total)

Add other labels like method="GET" or method="POST" to split by HTTP method while keeping a single metric name:

requests_total{path="/cart",method="GET"}
requests_total{path="/cart",method="POST"}

Internally the metric name is represented as a label named __name__. Example:

{__name__="node_cpu_seconds_total",cpu="0"}

Labels that begin and end with double underscores (e.g., __name__) are considered internal to Prometheus. Prometheus automatically adds at least two labels to scraped metrics: instance (the scraped target) and job (the scrape job name). Example metric with those automatic labels:

node_boot_time_seconds
node_boot_time_seconds{instance="192.168.1.168:9100",job="node"}

A minimal prometheus.yml job configuration that produces those labels:

job_name: "node"
scheme: https
basic_auth:
  username: prometheus
  password: password
static_configs:
  - targets: ["192.168.1.168:9100"]

Beware of high-cardinality labels (e.g., including user IDs, session IDs, or GUIDs). Each distinct label value creates a separate time series — too many series can severely impact Prometheus performance and storage.

Best practices (summary)

Choose stable, descriptive metric names.
Use labels for dimensions you will filter or aggregate by.
Avoid labels with unbounded cardinality (user IDs, request IDs).
Prefer histograms for cross-instance aggregation of distributions; use summaries when you need per-instance quantiles.
Use the # HELP and # TYPE comments to document metrics for humans and tools.

Links and references

Prometheus official docs: https://prometheus.io/docs/
Prometheus metrics and instrumenting applications: https://prometheus.io/docs/practices/instrumentation/

Introduction

Observability Fundamentals

Prometheus Fundamentals

PromQL

Dashboarding & Visualization

Application Instrumentation

Service Discovery

Push Gateway

Alerting

Monitoring Kubernetes

Conclusion

Scaling & Long Term Storage

Metric types

When to choose histogram vs. summary

Metric and label naming rules

Best practices (summary)

Links and references

Watch Video

​Metric types

​When to choose histogram vs. summary

​Metric and label naming rules

​Best practices (summary)

​Links and references

Watch Video

Metric types

When to choose histogram vs. summary

Metric and label naming rules

Best practices (summary)

Links and references