> ## Documentation Index
> Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Demo Metrics Collection with Prometheus

> Guide to using Prometheus and PromQL for Kubernetes monitoring, covering targets, metric types, label filtering, aggregation, rates and dashboard and alert query patterns

You cannot manage what you cannot measure. Prometheus is the de facto metrics standard for Kubernetes: Grafana visualizes Prometheus metrics, alerting rules evaluate against Prometheus data, and many operators expose Prometheus endpoints. Learning to query Prometheus with PromQL is essential for debugging, capacity planning, and platform observability.

In this lesson we'll:

* Verify our monitoring stack is running.
* Inspect Prometheus scrape targets.
* Cover PromQL fundamentals: gauges, counters, label filtering, aggregation, and calculating error rates.
* Show common query patterns used by dashboards and alerts.

## Check the monitoring stack

The monitoring stack is deployed into the `monitoring` namespace. Confirm the pods are running with kubectl:

```bash theme={null}
controlplane ~ ➜ kubectl get pods -n monitoring
NAME                                                           READY   STATUS    RESTARTS   AGE
alertmanager-kube-prometheus-stack-alertmanager-0              2/2     Running   0          20m
kube-prometheus-stack-kube-state-metrics-567d49447b-bjg7z     1/1     Running   0          20m
kube-prometheus-stack-operator-5d99bc6db8-xgr65                1/1     Running   0          20m
kube-prometheus-stack-prometheus-node-exporter-gjrz5          1/1     Running   0          20m
prometheus-kube-prometheus-stack-prometheus-0                 2/2     Running   0          20m

controlplane ~ ➜
```

Prometheus uses a pull model: it scrapes configured targets at a schedule that Prometheus controls. This centralizes discovery and scraping configuration compared with push-based agents.

## Prometheus targets

Open the Prometheus UI and inspect Status → Targets to see the endpoints Prometheus is scraping. If a target shows as DOWN here, Prometheus is not collecting data from it — this is the primary place to start when metrics are missing.

![A screenshot of the Prometheus web UI (Target health) showing a list of serviceMonitor targets with endpoints, label metadata, last scrape times, and green "UP" status indicators. The page displays multiple monitored endpoints (IP:port/metrics) and their scrape details.](https://mintcdn.com/kodekloud-c4ac6d9a/seqLTE3JlSrVAtw9/images/Prep-Course-Certified-Cloud-Native-Platform-Engineering-CNPE/Observability-and-Operations/Demo-Metrics-Collection-with-Prometheus/prometheus-servicemonitor-targets-up.jpg?fit=max\&auto=format\&n=seqLTE3JlSrVAtw9\&q=85\&s=54e9bab7f9bc1425acf0915df8f24392)

Example target details you will see under Targets (labels, endpoints, last scrape, state):

```text theme={null}
serviceMonitor/monitoring/kube-prometheus-stack-operator/0
Endpoint
https://172.17.0.6:10250/metrics
Labels
container="kube-prometheus-stack" endpoint="https" instance="172.17.0.6:10250" job="kube-prometheus-stack-operator"
namespace="monitoring" pod="kube-prometheus-stack-operator-5d99bc6db8-xgr65"
service="kube-prometheus-stack-operator"
Last scrape
11.551s ago
2ms
State
UP

... (additional serviceMonitor entries follow)
```

## Gauges vs Counters

Prometheus metrics come in different types. Two common ones are gauges and counters.

| Metric Type | Description | Example use |
| - | - | - |
| Gauge | Represents a value that can go up and down (current state) | `node_memory_MemAvailable_bytes` |
| Counter | A monotonically increasing value (cumulative) | `http_requests_total` |

Use gauges for instantaneous state (free memory, CPU usage) and counters for totals that you convert to rates or increases over windows.

## Gauges

A common gauge is `node_memory_MemAvailable_bytes`. Run this in the Prometheus expression field and convert bytes to gigabytes:

```promql theme={null}
node_memory_MemAvailable_bytes / 1024 / 1024 / 1024
```

Sample output (one series with label metadata and the numeric value):

```text theme={null}
{container="node-exporter", endpoint="http-metrics", instance="10.244.7.179:9100", job="node-exporter", namespace="monitoring", pod="kube-prometheus-stack-prometheus-node-exporter-gjrz5", service="kube-prometheus-stack-prometheus-node-exporter"} 48.814842224121094
```

This indicates approximately 48.8 GiB of available memory for that node-exporter instance.

## Counters and rates

Counters only increase, so the raw counter value is rarely meaningful by itself. Use `rate()` or `increase()` to interpret counters:

* `rate(<counter>[5m])` — per-second velocity estimated over the last 5 minutes.
* `increase(<counter>[5m])` — total increase over the last 5 minutes.

Raw counter example:

```promql theme={null}
apiserver_request_total
```

Per-second rate example:

```promql theme={null}
rate(apiserver_request_total[5m])
```

## Filtering by labels

Prometheus metrics carry labels (dimensions) you can match with label selectors inside `{}`. Use exact matches or regex with `=~`.

Example: rate of successful GET requests (HTTP 200) to the API server:

```promql theme={null}
rate(apiserver_request_total{verb="GET", code="200"}[5m])
```

Sample returned series (one series per unique combination of the remaining labels):

```text theme={null}
{code="200", endpoint="https", instance="10.244.7.179:6443", job="apiserver", namespace="default", subresource="/livez", verb="GET"} 0.2
{code="200", endpoint="https", instance="10.244.7.179:6443", job="apiserver", namespace="default", subresource="/readyz", verb="GET"} 1
...
```

## Aggregation

Aggregate across series using aggregation operators (sum, avg, max, min, etc.) and optionally group by labels.

Example: total rate per HTTP verb across all series:

```promql theme={null}
sum(rate(apiserver_request_total[5m])) by (verb)
```

Sample output:

```text theme={null}
{verb="GET"} 1.5555555555555556
{verb="LIST"} 0.011111111111111112
{verb="WATCH"} 0.45185185185185184
...
```

Note: the `by (label)` clause must follow the aggregation operator: `sum(...) by (label)`.

## Two common query patterns for dashboards

1. Totals over a window — useful for pie charts or distributions. Use `increase()` to get counts over a time interval.

Example: total HTTP requests per verb over the last 5 minutes:

```promql theme={null}
sum(increase(http_requests_total[5m])) by (verb)
```

Namespace-scoped totals over a 1 hour window, grouped by HTTP status code:

```promql theme={null}
sum(increase(http_requests_total{namespace="workloads"}[1h])) by (code)
```

2. Error rate percentage — useful for platform health dashboards and alerts.

Compute the percentage of requests that returned 5xx status codes over the last 5 minutes. Use a regex matcher to select 5xx responses:

```promql theme={null}
100 * sum(rate(apiserver_request_total{code=~"5.."}[5m])) / sum(rate(apiserver_request_total[5m]))
```

This returns the percentage of server errors in the chosen window. Spikes here indicate services returning 5xx responses and warrant investigation.

## Quick reference — common PromQL functions

| Function | Purpose | Example |
| - | - | - |
| `rate()` | Calculates per-second rate for counters over a window | `rate(apiserver_request_total[5m])` |
| `irate()` | Instantaneous per-second rate using the last two samples (use carefully) | `irate(http_requests_total[1m])` |
| `increase()` | Total increase of a counter over a time window | `increase(http_requests_total[5m])` |
| `sum(...) by (label)` | Aggregate and group by labels | `sum(rate(http_requests_total[5m])) by (code)` |
| `avg()` / `max()` / `min()` | Aggregation operators for summary statistics | `max(node_memory_MemAvailable_bytes)` |

## Troubleshooting tips

* Inspect Status → Targets in the Prometheus UI first when metrics are missing — DOWN targets indicate no scraping.
* Use label filtering to narrow noisy results and then aggregate.
* Prefer `rate()` or `increase()` for counters; use `gauge` raw values directly for instantaneous state.

When writing PromQL in MDX/MD files, always place expressions containing curly braces and angle brackets in code blocks or inline code (backticks) so the MDX parser does not try to interpret them as JSX or JS objects.

## Links and references

* Prometheus documentation: [https://prometheus.io/docs/](https://prometheus.io/docs/)
* PromQL quick reference: [https://prometheus.io/docs/prometheus/latest/querying/basics/](https://prometheus.io/docs/prometheus/latest/querying/basics/)
* Grafana: [https://grafana.com/](https://grafana.com/)

Practice these queries directly in the Prometheus UI to reinforce the concepts and build reusable dashboard panels and alerts.

> **[Watch Video](https://learn.kodekloud.com/user/courses/prep-course-certified-cloud-native-platform-engineer-cnpe/module/9bd090c8-8d99-4742-b50c-ae63e516e6b9/lesson/f0820e36-3f8c-4741-8b79-6dbe9c077761)**

> **[Practice Lab](https://learn.kodekloud.com/user/courses/prep-course-certified-cloud-native-platform-engineer-cnpe/module/9bd090c8-8d99-4742-b50c-ae63e516e6b9/lesson/c31eb94b-7f3b-49a3-909e-5fda15273fc2)**


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.