> ## Documentation Index
> Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Metrics Architecture Collection Storage and Querying

> Prometheus metrics architecture, collection, metric types, labels, and PromQL for unified observability, querying, dashboards, and alerting

Metrics are the foundational pillar of observability — they answer "what is happening" in your system by providing quantitative, time-series data. Collecting metrics at scale and making them actionable are separate problems: collection requires reliable discovery and ingestion, while usefulness depends on consistent naming, labeling, and correct querying.

In this lesson you’ll learn:

* A unified metrics collection architecture and why it matters
* How Prometheus uses a pull-based model for scraping and detecting health
* The three primary Prometheus metric types and which PromQL functions to use
* How labels make metrics dimensional and enable powerful aggregations
* Enough PromQL to build Grafana dashboards and alerts

![A slide titled "Learning Objectives" with a colorful vertical numbered list. The items cover unified metrics architecture, Prometheus's pull-based model, counters/gauges/histograms, using labels, and basic PromQL queries.](https://mintcdn.com/kodekloud-c4ac6d9a/seqLTE3JlSrVAtw9/images/Prep-Course-Certified-Cloud-Native-Platform-Engineering-CNPE/Observability-and-Operations/Metrics-Architecture-Collection-Storage-and-Querying/prometheus-metrics-architecture-pull-labels-promql.jpg?fit=max\&auto=format\&n=seqLTE3JlSrVAtw9\&q=85\&s=f88442e55a74a32e6c6b5055171bdde5)

Real-world example: why consistent labels matter

A logistics company ran sixty microservices across three Kubernetes clusters. Each team used their own metric names and labeling conventions. During a database connection-pool exhaustion incident, the platform team needed to correlate API error rates with DB connection counts and pod memory usage. The data existed across three systems with different time granularities and label sets, so joining them was impossible — the correlation that should have taken minutes turned into a 90-minute spreadsheet merge.

After migrating to Prometheus with consistent metric names and labels, the same correlation became a single-line PromQL query and returned results instantly. Consistent labels are not just convenience — they enable fast, correct incident response.

The fragmentation problem and the Prometheus solution

In a typical Kubernetes platform you’ll see many metric sources: application `/metrics` endpoints, node exporters, cAdvisor, kube-state-metrics, and external systems. Without a unified collection approach you get:

* No single source of truth
* Formatting fragmentation and inconsistent label sets
* Limited ability to correlate metrics across components
* Multiple dashboards showing different views of the same incident

Prometheus addresses this by providing a single scraping and storage system (TSDB) with a powerful query language (PromQL). Rather than collecting more disparate data, focus on a unified collection architecture and consistent labeling.

Prometheus architecture at a glance

Prometheus uses a pull-based model:

* Targets expose an HTTP `/metrics` endpoint.
* Service discovery finds targets (Kubernetes SD, file\_sd, DNS, etc.).
* Prometheus scrapes those endpoints at configured intervals.
* Scraped samples are stored in Prometheus’ time-series database (TSDB).
* PromQL queries the TSDB for dashboards, alerts, and analysis.

This flow gives central control and built-in health detection: when Prometheus can’t scrape a target, the target’s absence becomes observable.

![A diagram titled "Prometheus: Pull-Based Collection" showing stacked colored boxes that outline Prometheus components and workflow. It lists Scrape Targets (pods/nodes/exporters), Service Discovery (Kubernetes SD), Time Series Database (TSDB), and PromQL → Grafana dashboards/alerting rules.](https://mintcdn.com/kodekloud-c4ac6d9a/seqLTE3JlSrVAtw9/images/Prep-Course-Certified-Cloud-Native-Platform-Engineering-CNPE/Observability-and-Operations/Metrics-Architecture-Collection-Storage-and-Querying/prometheus-pull-collection-diagram.jpg?fit=max\&auto=format\&n=seqLTE3JlSrVAtw9\&q=85\&s=6887c2633cbbaa23163491b50a62ad2a)

Why pull (scrape) instead of push?

Two main advantages of Prometheus’ pull model:

1. Central control — a single place controls what gets scraped and when, simplifying configuration and rate limits.
2. Health detection — failed scrapes are recorded as missing samples. Debugging is as simple as curling a target’s `/metrics` endpoint to compare what Prometheus scraped.

Metric types and correct query patterns

Choosing the right metric type and PromQL function is crucial. Applying the wrong function to a metric is a common source of errors.

Use this reference table to match metric types to the correct query patterns:

| Metric Type | Description | Prometheus naming examples | How to query |
| - | - | -: | - |
| Counter | Monotonic, only increases (resets on restart) | `http_requests_total` | Use `rate()` or `increase()` over a range vector: `rate(http_requests_total[5m])` |
| Gauge | Instantaneous value that can go up or down | `node_memory_available_bytes` | Read directly (instant vector): `node_memory_available_bytes{job="node"}` |
| Histogram | Observations bucketed (latency/sizes) | `http_request_duration_seconds_bucket` | Use `histogram_quantile()` on `_bucket` series to compute percentiles |

Illustrative sequences:

```text theme={null}
# Counter (monotonic)
0 → 10 → 25 → 47 → 103 →
http_requests_total

# Gauge (instantaneous)
4.2 → 3.8 → 5.1 → 2.9 → 4.5
node_memory_available_bytes

# Histogram buckets (counts)
<0.1s: 500  <0.5s: 120  <1s: 30
http_request_duration_seconds_bucket
```

Key PromQL functions to remember:

* `rate()` — convert counter samples into per-second rates over a time window
* `increase()` — compute total increments over a window
* `histogram_quantile()` — derive percentiles from histogram buckets
* Aggregators: `sum()`, `avg()`, `max()`, etc.

Tip: Applying `rate()` to a gauge is meaningless (gauges can move up and down). Conversely, reading a counter’s raw value is usually meaningless — convert it to a rate or delta first.

Warning: Histograms require correct instrumentation. Avoid attempting percentiles from manually aggregated values — use Prometheus histograms and `histogram_quantile()` instead.

Labels: dimensional metrics

Labels transform single metrics into multi-dimensional datasets. Without labels, you’d need separate metric names per dimension, which is unsustainable:

```text theme={null}
checkout_requests_total  1420
payment_requests_total   980
checkout_errors_total    42
payment_errors_total     18
```

With labels you can express the same data compactly and perform flexible queries:

```Prometheus theme={null}
http_requests_total{service="checkout",status="200"}
http_requests_total{service="payment",status="500"}
```

Labels let you:

* Filter to a subset (e.g., `namespace="production"`)
* Group and aggregate (e.g., `sum by (service) (...)`)
* Create targeted alerts (e.g., `service="checkout", severity="critical"`)

Kubernetes injects common labels (for example, `namespace`, `pod`, `container`) when using the appropriate exporters and the Prometheus Operator, making namespace- or pod-level queries straightforward.

PromQL basics — three common patterns

PromQL can appear complex, but three patterns cover most needs:

1. Instant vectors — retrieve the current value of a metric (optionally filtered by labels).
2. Range vectors + functions — apply functions like `rate()` or `increase()` to counters over a time window.
3. Aggregation — roll up series with `sum by (...)`, `avg by (...)`, `max by (...)`.

Examples:

```promql theme={null}
# Current memory per pod (instant vector)
container_memory_usage_bytes{namespace="production"}

# Requests per second (5m window) — counter wrapped in rate()
rate(http_requests_total[5m])

# Total 5xx error rate by service (aggregate and filter)
sum by (service) (
  rate(http_requests_total{code=~"5.."}[5m])
)
```

A few more useful snippets:

```promql theme={null}
# Per-pod CPU usage (instant, seconds)
container_cpu_usage_seconds_total{pod="web-1234"} 

# 99th percentile request duration by service (histogram-based)
histogram_quantile(0.99, sum by (le, service) (rate(http_request_duration_seconds_bucket[5m])))

# Increase in errors over last 1h
increase(my_app_errors_total[1h])
```

How Prometheus discovers targets in Kubernetes

Prometheus supports multiple discovery methods in Kubernetes:

* Pod annotations (quick and simple): add annotations so Kubernetes SD picks them up. Example pod annotation:

```yaml theme={null}
metadata:
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "8080"
```

* ServiceMonitor CRDs (recommended for production): provided by the Prometheus Operator; `ServiceMonitor` resources declaratively define scrape targets for Services and are much easier to manage at scale.
* Static config / file\_sd: useful for external systems and legacy services outside the cluster.

Common metric sources

The standard ecosystem includes:

* kube-state-metrics — exposes the state of Kubernetes objects (Deployments, ReplicaSets, Pods)
* node-exporter — node-level hardware and OS metrics (CPU, disk, network)
* cAdvisor / kubelet metrics — container-level CPU and memory stats
* Application `/metrics` endpoints — custom business and application metrics

Examples of what each provides:

| Source | Typical content |
| - | - |
| kube-state-metrics | Deployment replica counts, pod statuses |
| node-exporter | Host-level CPU, memory, filesystem, network |
| cAdvisor / kubelet | Per-container CPU/memory usage |
| App `/metrics` | Request counts, latencies, business metrics |

Links and further reading

* Prometheus documentation: [https://prometheus.io/docs/](https://prometheus.io/docs/)
* PromQL basics: [https://prometheus.io/docs/prometheus/latest/querying/basics/](https://prometheus.io/docs/prometheus/latest/querying/basics/)
* Prometheus Operator / ServiceMonitor: [https://github.com/prometheus-operator/prometheus-operator](https://github.com/prometheus-operator/prometheus-operator)
* Kubernetes documentation: [https://kubernetes.io/docs/](https://kubernetes.io/docs/)

Summary — what to practice

* Design a single collection architecture and enforce consistent metric names and labels.
* Remember the three metric types: counters, gauges, histograms — and which PromQL functions to use for each.
* Prefer pull-based scraping in Kubernetes for central control and health visibility.
* Use labels to make metrics dimensional and powerful for aggregation and alerting.
* Practice common PromQL patterns: instant vectors, `rate()` on counters, and aggregation by labels.

![A slide titled "Key Takeaways" with four numbered points down the right side. The points summarize Prometheus concepts: it's pull-based, has three metric/query patterns, uses labels for dimensional metrics, and uses PromQL for queries.](https://mintcdn.com/kodekloud-c4ac6d9a/seqLTE3JlSrVAtw9/images/Prep-Course-Certified-Cloud-Native-Platform-Engineering-CNPE/Observability-and-Operations/Metrics-Architecture-Collection-Storage-and-Querying/prometheus-takeaways-pull-patterns-labels-promql.jpg?fit=max\&auto=format\&n=seqLTE3JlSrVAtw9\&q=85\&s=13152576ecf44443fb577bc0f8eaf287)

> **[Watch Video](https://learn.kodekloud.com/user/courses/prep-course-certified-cloud-native-platform-engineer-cnpe/module/9bd090c8-8d99-4742-b50c-ae63e516e6b9/lesson/dafd2315-36be-4318-b3e4-cc408da620c5)**


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.