Skip to main content
Metrics are the foundational pillar of observability — they answer “what is happening” in your system by providing quantitative, time-series data. Collecting metrics at scale and making them actionable are separate problems: collection requires reliable discovery and ingestion, while usefulness depends on consistent naming, labeling, and correct querying. In this lesson you’ll learn:
  • A unified metrics collection architecture and why it matters
  • How Prometheus uses a pull-based model for scraping and detecting health
  • The three primary Prometheus metric types and which PromQL functions to use
  • How labels make metrics dimensional and enable powerful aggregations
  • Enough PromQL to build Grafana dashboards and alerts
A slide titled "Learning Objectives" with a colorful vertical numbered list. The items cover unified metrics architecture, Prometheus's pull-based model, counters/gauges/histograms, using labels, and basic PromQL queries.
Real-world example: why consistent labels matter A logistics company ran sixty microservices across three Kubernetes clusters. Each team used their own metric names and labeling conventions. During a database connection-pool exhaustion incident, the platform team needed to correlate API error rates with DB connection counts and pod memory usage. The data existed across three systems with different time granularities and label sets, so joining them was impossible — the correlation that should have taken minutes turned into a 90-minute spreadsheet merge. After migrating to Prometheus with consistent metric names and labels, the same correlation became a single-line PromQL query and returned results instantly. Consistent labels are not just convenience — they enable fast, correct incident response. The fragmentation problem and the Prometheus solution In a typical Kubernetes platform you’ll see many metric sources: application /metrics endpoints, node exporters, cAdvisor, kube-state-metrics, and external systems. Without a unified collection approach you get:
  • No single source of truth
  • Formatting fragmentation and inconsistent label sets
  • Limited ability to correlate metrics across components
  • Multiple dashboards showing different views of the same incident
Prometheus addresses this by providing a single scraping and storage system (TSDB) with a powerful query language (PromQL). Rather than collecting more disparate data, focus on a unified collection architecture and consistent labeling. Prometheus architecture at a glance Prometheus uses a pull-based model:
  • Targets expose an HTTP /metrics endpoint.
  • Service discovery finds targets (Kubernetes SD, file_sd, DNS, etc.).
  • Prometheus scrapes those endpoints at configured intervals.
  • Scraped samples are stored in Prometheus’ time-series database (TSDB).
  • PromQL queries the TSDB for dashboards, alerts, and analysis.
This flow gives central control and built-in health detection: when Prometheus can’t scrape a target, the target’s absence becomes observable.
A diagram titled "Prometheus: Pull-Based Collection" showing stacked colored boxes that outline Prometheus components and workflow. It lists Scrape Targets (pods/nodes/exporters), Service Discovery (Kubernetes SD), Time Series Database (TSDB), and PromQL → Grafana dashboards/alerting rules.
Why pull (scrape) instead of push? Two main advantages of Prometheus’ pull model:
  1. Central control — a single place controls what gets scraped and when, simplifying configuration and rate limits.
  2. Health detection — failed scrapes are recorded as missing samples. Debugging is as simple as curling a target’s /metrics endpoint to compare what Prometheus scraped.
Metric types and correct query patterns Choosing the right metric type and PromQL function is crucial. Applying the wrong function to a metric is a common source of errors. Use this reference table to match metric types to the correct query patterns: Illustrative sequences:
Key PromQL functions to remember:
  • rate() — convert counter samples into per-second rates over a time window
  • increase() — compute total increments over a window
  • histogram_quantile() — derive percentiles from histogram buckets
  • Aggregators: sum(), avg(), max(), etc.
Tip: Applying rate() to a gauge is meaningless (gauges can move up and down). Conversely, reading a counter’s raw value is usually meaningless — convert it to a rate or delta first.
Warning: Histograms require correct instrumentation. Avoid attempting percentiles from manually aggregated values — use Prometheus histograms and histogram_quantile() instead.
Labels: dimensional metrics Labels transform single metrics into multi-dimensional datasets. Without labels, you’d need separate metric names per dimension, which is unsustainable:
With labels you can express the same data compactly and perform flexible queries:
Labels let you:
  • Filter to a subset (e.g., namespace="production")
  • Group and aggregate (e.g., sum by (service) (...))
  • Create targeted alerts (e.g., service="checkout", severity="critical")
Kubernetes injects common labels (for example, namespace, pod, container) when using the appropriate exporters and the Prometheus Operator, making namespace- or pod-level queries straightforward. PromQL basics — three common patterns PromQL can appear complex, but three patterns cover most needs:
  1. Instant vectors — retrieve the current value of a metric (optionally filtered by labels).
  2. Range vectors + functions — apply functions like rate() or increase() to counters over a time window.
  3. Aggregation — roll up series with sum by (...), avg by (...), max by (...).
Examples:
A few more useful snippets:
How Prometheus discovers targets in Kubernetes Prometheus supports multiple discovery methods in Kubernetes:
  • Pod annotations (quick and simple): add annotations so Kubernetes SD picks them up. Example pod annotation:
  • ServiceMonitor CRDs (recommended for production): provided by the Prometheus Operator; ServiceMonitor resources declaratively define scrape targets for Services and are much easier to manage at scale.
  • Static config / file_sd: useful for external systems and legacy services outside the cluster.
Common metric sources The standard ecosystem includes:
  • kube-state-metrics — exposes the state of Kubernetes objects (Deployments, ReplicaSets, Pods)
  • node-exporter — node-level hardware and OS metrics (CPU, disk, network)
  • cAdvisor / kubelet metrics — container-level CPU and memory stats
  • Application /metrics endpoints — custom business and application metrics
Examples of what each provides: Links and further reading Summary — what to practice
  • Design a single collection architecture and enforce consistent metric names and labels.
  • Remember the three metric types: counters, gauges, histograms — and which PromQL functions to use for each.
  • Prefer pull-based scraping in Kubernetes for central control and health visibility.
  • Use labels to make metrics dimensional and powerful for aggregation and alerting.
  • Practice common PromQL patterns: instant vectors, rate() on counters, and aggregation by labels.
A slide titled "Key Takeaways" with four numbered points down the right side. The points summarize Prometheus concepts: it's pull-based, has three metric/query patterns, uses labels for dimensional metrics, and uses PromQL for queries.

Watch Video