> ## Documentation Index > Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt > Use this file to discover all available pages before exploring further. # Metrics > Explains Prometheus metrics structure, time series, metric types, labels and metadata with guidance on instrumentation and avoiding high cardinality In this lesson we’ll explain how Prometheus metrics are structured, how time series are formed, and how to interpret different metric types. You’ll learn how labels and metric names work, the meaning of `# HELP` and `# TYPE` metadata, and practical guidance for avoiding high-cardinality issues. A Prometheus metric has three main parts: * a descriptive name, * a set of labels (key/value pairs), * and a numeric value recorded at a specific time. The text exposition format looks like: ```text theme={null} {,} ``` Example from the Node Exporter: ```text theme={null} node_cpu_seconds_total{cpu="0",mode="idle"} 258277.86 ``` This indicates that CPU 0 has accumulated 258,277.86 seconds in the "idle" state. If a machine has multiple CPUs, the same metric appears multiple times with different `cpu` labels: ```text theme={null} node_cpu_seconds_total{cpu="0",mode="idle"} 258277.86 node_cpu_seconds_total{cpu="1",mode="idle"} 427262.54 node_cpu_seconds_total{cpu="2",mode="idle"} 283288.12 node_cpu_seconds_total{cpu="3",mode="idle"} 258202.33 ``` Each unique metric name + label set identifies a separate stream of timestamped values (a time series). When Prometheus scrapes a target it records the sample timestamp (by default the scrape time) as a Unix timestamp (seconds since the epoch) for samples without an explicit timestamp. For example: ```text theme={null} 1668215300 ``` You can convert Unix timestamps using any Unix timestamp converter; dashboards typically render these in local time zones automatically. Stored samples therefore include a value plus a timestamp, e.g.: ```text theme={null} node_cpu_seconds_total{cpu="0",mode="idle"} 258277.86 1668215300 ``` The image is a diagram titled "Counter," highlighting metrics like "Total # requests," "Total # Exceptions," and "Total # of job executions," showing how these numbers can only increase. ## Time series A "time series" in Prometheus is a stream of timestamped values that share the same metric name and identical label sets (same keys and values). In other words: metric name + labels = time series. Examples: ```text theme={null} node_filesystem_files{device="sda2", instance="server1"} node_filesystem_files{device="sda3", instance="server1"} node_filesystem_files{device="sda2", instance="server2"} node_filesystem_files{device="sda3", instance="server2"} node_cpu_seconds_total{cpu="0", instance="server1"} node_cpu_seconds_total{cpu="1", instance="server1"} node_cpu_seconds_total{cpu="0", instance="server2"} node_cpu_seconds_total{cpu="1", instance="server2"} ``` Although there are only two metric names above, the different label combinations produce eight distinct time series. As Prometheus scrapes these targets periodically (for example every 15 or 30 seconds), it appends new timestamped values to each time series. ## Metric metadata: HELP and TYPE Prometheus text exposition may include two helpful metadata comments per metric: * `# HELP` — a short human-readable description of the metric. * `# TYPE` — the metric type (`counter`, `gauge`, `histogram`, `summary`). Example: ```text theme={null} # HELP node_disk_discard_time_seconds_total This is the total number of seconds spent by all discards. # TYPE node_disk_discard_time_seconds_total counter node_disk_discard_time_seconds_total{device="sda"} 0 node_disk_discard_time_seconds_total{device="sr0"} 0 ``` Including `# HELP` and `# TYPE` in your exporters makes metrics easier to understand and reduces ambiguity when other engineers query or consume them. ## Metric types Prometheus defines four primary metric types. Here’s a concise reference you can use when designing instrumentation. | Type | Description | Typical use cases | Example | | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------- | ------------------------------------------------------- | | Counter | Cumulative metric that only increases (or resets on restart). | Total requests, error counts, job executions. | `node_cpu_seconds_total{cpu="0",mode="idle"} 258277.86` | | Gauge | Value that can go up and down. | Current memory usage, temperature, concurrent connections. | `process_resident_memory_bytes 1.23e+07` | | Histogram | Observations grouped into configurable cumulative buckets; also exports a `_count` and `_sum`. Useful for aggregating distributions across instances. | Response time distributions, payload sizes. | `http_request_duration_seconds_bucket{le="0.5"} 240` | | Summary | Tracks event counts, sum, and client-side computed quantiles (percentiles). Quantiles are computed per-instance and cannot be aggregated reliably across instances. | Latency percentiles on a single instance. | `http_request_duration_seconds{quantile="0.5"} 0.12` | Histograms collect counts for bucket boundaries (buckets are cumulative). For example, with buckets 0.2s, 0.5s, 1s you get counts of requests that completed in less than 0.2s, less than 0.5s (including those under 0.2s), and less than 1s (including those under 0.5s): The image is an infographic about histograms, displaying response time and request size metrics along with a bar chart illustrating response time buckets. Summaries report quantiles directly (for example 20%, 50%, 80%) and are calculated on the client side. Because quantiles in summaries are per-instance they are not directly aggregatable across multiple instances the way histogram bucket counts are. The image is a summary slide comparing histogram-like data tracking, showing percentiles for response time and request size, with a bar chart illustrating response time percentiles. ## Metric names and labels Metric names should be descriptive and follow Prometheus naming conventions: ASCII letters, numbers, underscores, and colons. Avoid colons in custom metrics — they are typically reserved for recording rules. Labels are key/value pairs that further qualify a metric and let you slice and group metrics by attributes such as `path`, `method`, or `instance`. Label names may include ASCII letters, numbers, and underscores. Why labels are useful — a brief example: A naive approach would create separate metrics per endpoint: ```text theme={null} requests_auth_total requests_user_total requests_cart_total ``` This approach makes it hard to compute totals without knowing each metric name. A better approach uses a single metric with a `path` label: ```text theme={null} requests_total{path="/auth"} 123 requests_total{path="/user"} 456 requests_total{path="/cart"} 78 ``` You can then compute totals or grouped sums with PromQL: ```promql theme={null} sum(requests_total) sum(requests_total) by (path) ``` Add other labels as needed (for example `method="GET"`). Each unique combination of label keys and values creates a distinct time series. The image illustrates a comparison of two methods for calculating total API requests in an e-commerce app, highlighting the difficulty of calculating total requests across all paths versus summing requests using a specific method with labeled paths. Internally, Prometheus stores the metric name as a label called `__name__`. Example: ```text theme={null} node_cpu_seconds_total{cpu="0"} = {__name__="node_cpu_seconds_total", cpu="0"} ``` Labels that begin and end with double underscores (for example `__name__`) are internal to Prometheus. Prometheus also automatically assigns two labels to every scraped metric: * `instance` — the target address that was scraped. * `job` — the scrape job name from your Prometheus configuration. Example scrape job in `prometheus.yml`: ```yaml theme={null} scrape_configs: - job_name: "node" scheme: https basic_auth: username: prometheus password: password static_configs: - targets: - "192.168.1.168:9100" ``` Avoid high-cardinality labels (for example, user IDs, session IDs, or highly variable full request paths). Every distinct label value combination creates a new time series, which can quickly exhaust memory and storage in Prometheus. ## Quick references and further reading * Prometheus data model and time series: [Prometheus concepts — data model](https://prometheus.io/docs/concepts/data_model/) * Node Exporter guide: [Node Exporter](https://prometheus.io/docs/guides/node-exporter/) * Convert Unix timestamps: [Epoch Converter](https://www.epochconverter.com/) Keep these practical rules in mind: * Use meaningful metric names. * Favor labels over separate metric names when the distinguishing characteristic is a dimension (e.g., `path`, `method`). * Choose histograms if you need to aggregate distributions across instances; use summaries for per-instance quantiles when aggregation across instances is not required. * Design labels to keep cardinality manageable.