Skip to main content
You cannot manage what you cannot measure. Prometheus is the de facto metrics standard for Kubernetes: Grafana visualizes Prometheus metrics, alerting rules evaluate against Prometheus data, and many operators expose Prometheus endpoints. Learning to query Prometheus with PromQL is essential for debugging, capacity planning, and platform observability. In this lesson we’ll:
  • Verify our monitoring stack is running.
  • Inspect Prometheus scrape targets.
  • Cover PromQL fundamentals: gauges, counters, label filtering, aggregation, and calculating error rates.
  • Show common query patterns used by dashboards and alerts.

​
Check the monitoring stack

The monitoring stack is deployed into the monitoring namespace. Confirm the pods are running with kubectl:
Prometheus uses a pull model: it scrapes configured targets at a schedule that Prometheus controls. This centralizes discovery and scraping configuration compared with push-based agents.

​
Prometheus targets

Open the Prometheus UI and inspect Status → Targets to see the endpoints Prometheus is scraping. If a target shows as DOWN here, Prometheus is not collecting data from it — this is the primary place to start when metrics are missing.
A screenshot of the Prometheus web UI (Target health) showing a list of serviceMonitor targets with endpoints, label metadata, last scrape times, and green "UP" status indicators. The page displays multiple monitored endpoints (IP:port/metrics) and their scrape details.
Example target details you will see under Targets (labels, endpoints, last scrape, state):

​
Gauges vs Counters

Prometheus metrics come in different types. Two common ones are gauges and counters. Use gauges for instantaneous state (free memory, CPU usage) and counters for totals that you convert to rates or increases over windows.

​
Gauges

A common gauge is node_memory_MemAvailable_bytes. Run this in the Prometheus expression field and convert bytes to gigabytes:
Sample output (one series with label metadata and the numeric value):
This indicates approximately 48.8 GiB of available memory for that node-exporter instance.

​
Counters and rates

Counters only increase, so the raw counter value is rarely meaningful by itself. Use rate() or increase() to interpret counters:
  • rate(<counter>[5m]) — per-second velocity estimated over the last 5 minutes.
  • increase(<counter>[5m]) — total increase over the last 5 minutes.
Raw counter example:
Per-second rate example:

​
Filtering by labels

Prometheus metrics carry labels (dimensions) you can match with label selectors inside {}. Use exact matches or regex with =~. Example: rate of successful GET requests (HTTP 200) to the API server:
Sample returned series (one series per unique combination of the remaining labels):

​
Aggregation

Aggregate across series using aggregation operators (sum, avg, max, min, etc.) and optionally group by labels. Example: total rate per HTTP verb across all series:
Sample output:
Note: the by (label) clause must follow the aggregation operator: sum(...) by (label).

​
Two common query patterns for dashboards

  1. Totals over a window — useful for pie charts or distributions. Use increase() to get counts over a time interval.
Example: total HTTP requests per verb over the last 5 minutes:
Namespace-scoped totals over a 1 hour window, grouped by HTTP status code:
  1. Error rate percentage — useful for platform health dashboards and alerts.
Compute the percentage of requests that returned 5xx status codes over the last 5 minutes. Use a regex matcher to select 5xx responses:
This returns the percentage of server errors in the chosen window. Spikes here indicate services returning 5xx responses and warrant investigation.

​
Quick reference — common PromQL functions

​
Troubleshooting tips

  • Inspect Status → Targets in the Prometheus UI first when metrics are missing — DOWN targets indicate no scraping.
  • Use label filtering to narrow noisy results and then aggregate.
  • Prefer rate() or increase() for counters; use gauge raw values directly for instantaneous state.
When writing PromQL in MDX/MD files, always place expressions containing curly braces and angle brackets in code blocks or inline code (backticks) so the MDX parser does not try to interpret them as JSX or JS objects.
Practice these queries directly in the Prometheus UI to reinforce the concepts and build reusable dashboard panels and alerts.

Watch Video

Practice Lab