Skip to main content
You cannot manage what you cannot measure. Prometheus is the de facto metrics standard for Kubernetes. Most tools in the ecosystem integrate with Prometheus — Grafana visualizes its metrics, alerting systems trigger from them, and platform operators rely on PromQL to build meaningful dashboards and alerts. This guide walks through verifying a deployed monitoring stack, understanding Prometheus’ scrape model, and using common PromQL patterns for dashboards and troubleshooting.

​
Verify the monitoring stack

Confirm the monitoring namespace pods are running:
Prometheus uses a pull model: it periodically scrapes configured endpoints rather than relying on agents pushing metrics. This centralizes control over scrape frequency, relabeling, and the exact endpoints collected — applications only need to expose metrics.
Prometheus’ pull model gives central control over what and when to scrape. You control scrape intervals, relabeling, and which endpoints are collected — applications only need to expose metrics.
Open the Prometheus UI and navigate to Status → Targets to inspect which endpoints are being scraped and whether they are healthy.
The image shows a Prometheus monitoring dashboard displaying the status and health of various service monitors, each with endpoints, labels, and their current state.
If a target appears as DOWN in the Targets view, Prometheus is not collecting from it — begin troubleshooting from that page (network/DNS, service ports, pod readiness, or relabel rules). When targets are UP, use the Query tab to validate metrics and build expressions.
The image shows a Prometheus dashboard with a list of monitored service endpoints, each displaying labels, endpoints, last scrape times, and their current states as "UP".

​
Querying metrics: Gauges vs Counters

Use the Query tab to inspect raw metrics, test PromQL, and preview series for dashboard panels.
  • Gauges represent current states and can go up or down (example: available memory).
  • Counters only increase (monotonic) and represent totals over time (example: number of HTTP requests since process start).

​
Gauges — example (available memory)

Raw metric:
A sample returned value may look like:
Convert bytes to gibibytes for readability:
This returns values like 48.8148, representing ~48.8 GiB.

​
Counters — example (request rates)

Counters accumulate. To measure velocity use rate() (or irate() for instant rate) over a range vector. For example, requests per second over the last 5 minutes:
Filter by labels (dimensions) to narrow results, e.g., successful GET requests:
The Prometheus query editor offers metric and label suggestions as you type, aiding discovery.
The image shows a Prometheus query editor interface with suggestions for querying apiserver_request_total, a counter for API server requests. Various query metrics are displayed as the user prepares a query.

​
Aggregation patterns for dashboards

Two common aggregation patterns used in platform dashboards:
  1. Distribution (e.g., requests by verb or status code)
  2. Error rate percentage (ratio of error requests to total requests)
Useful PromQL functions and examples: Note: In table examples, label selectors and brace syntax are shown inline in backticks to avoid MDX parsing issues. Examples:
  • Per-verb request rate aggregated across instances:
  • Total requests over the last hour, grouped by HTTP status code (useful for pie charts or totals):
Use rate() (or irate() for instant rate) with counters when you want a per-second velocity. Use increase() when you want the total increment of a counter over a time window (useful for totals and pie charts).

​
Error rate percentage

A common SRE/platform metric is the percentage of requests that resulted in 5xx responses over a short window. Compute it as:
This returns the percentage of requests that had 5xx status codes in the last 5 minutes. Spikes in this metric typically indicate application or infrastructure issues that need investigation.

​
Quick troubleshooting checklist

  • Check Prometheus → Status → Targets first when metrics are missing.
  • Confirm pods and endpoints are Ready and reachable from the Prometheus server.
  • Review ServiceMonitor/PodMonitor relabel configs for unintended label drops.
  • Use the Query tab to validate that the expected metric names and labels exist.
  • Convert raw units (bytes, nanoseconds) into human-friendly units in queries for dashboards.

​
Summary

  • Prometheus scrapes targets (pull model) — verify targets in Status → Targets if metrics are missing.
  • Gauges: read current state; convert units for readability.
  • Counters: use rate()/irate() for velocity and increase() for totals.
  • Filter using label selectors (e.g., {verb="GET", code="200"}) and aggregate with sum(...) by (...) for dashboard-friendly series.
Now start practicing these Prometheus queries with hands-on exercises to get comfortable building dashboards and alerts.

Watch Video

Practice Lab