Skip to main content
Prometheus handles metric collection and Alertmanager handles alerting. This lesson focuses on the visualization layer: Grafana. Dashboards are how you triage incidents, spot trends, and communicate platform health. Grafana turns raw Prometheus metrics into focused, actionable visualizations. By the end of this lesson you will know how to connect Grafana to Prometheus, choose the correct panel types for different questions, and structure dashboards so you can triage incidents in seconds.
A presentation slide with a blue gradient panel on the left titled "Learning Objectives." On the right are two numbered goals: understanding Grafana's role as a visualization layer for Prometheus, and configuring a Prometheus datasource in Grafana.
Why visualization matters Imagine a flash sale where latency spikes. An on-call engineer typing PromQL manually in the Prometheus UI spends thirty seconds per query and another ten seconds to iterate — after fifteen minutes the root cause is still unclear. A colleague opens a pre-built Grafana dashboard and identifies the root cause in ninety seconds: a downstream payment service exhausted its connection pool. After that incident, Grafana dashboards became the mandatory first stop. Mean time to identify dropped from twenty minutes to under three minutes. Raw Prometheus output is often dense and not human-friendly. Grafana converts raw metrics into panels that immediately answer operational questions like:
  • Is the platform healthy?
  • Which services have increasing error rates?
  • Are we approaching resource limits?
How Grafana is structured Grafana is organized in three logical layers:
  • Data sources — connectors to metric/trace/log stores (Prometheus, Loki, Jaeger, CloudWatch, etc.).
  • Dashboards — collections of panels focused on a specific concern.
  • Panels — individual visualizations; each embeds a PromQL query and a chart/stat/table type.
A slide titled "Prometheus in Kubernetes" showing a layered diagram of monitoring components. It lists Datasources (Prometheus, Loki, Jaeger, CloudWatch, InfluxDB, etc.), Dashboards (collections of panels), and Panels (individual visualizations like time-series graphs, gauges, tables).
Connecting Grafana to Prometheus Step-by-step: configure Prometheus as a Grafana data source.
  1. In Grafana, go to Configuration → Data Sources → Add data source.
  2. Select “Prometheus”.
  3. Set the URL to your in-cluster Prometheus service address (point to the Kubernetes Service, not localhost or an external IP). Example values:
  1. Toggle “Default” so new panels use this data source by default.
  2. Click Save & Test to verify Grafana can reach Prometheus.
If you need to discover the Prometheus service name in Kubernetes:
Grafana can also be provisioned automatically with a datasource YAML for Infrastructure-as-Code deployments. Example provisioning file:
If Grafana and Prometheus are deployed inside the same Kubernetes cluster, prefer the in-cluster Service DNS name (e.g., prometheus or prometheus-server) and port 9090. Using localhost is a common misconfiguration when Grafana runs as a separate pod.
Three common pitfalls
Using an external Prometheus endpoint without proper network rules or authentication will cause intermittent failures. Ensure network policies, service discovery, and any reverse proxies are configured before relying on the datasource in production dashboards.
Choosing the right panel for the question Match visualization type to the operational question you need to answer. The table below maps common questions to panel types and gives example PromQL. Tips:
  • Use rate() for counters and avg()/sum() with by(...) for aggregations.
  • Avoid very long PromQL expressions in dashboards; encapsulate complex logic in recording rules (Prometheus) where possible.
Organizing dashboards (top-down triage pattern) A consistent layout dramatically speeds incident triage. Use a top-down flow so anyone opening the dashboard quickly gets to answers. Recommended rows and purpose:
  • Row 1 — Status overview: Stat panels that answer “Is everything broken?” at a glance.
  • Row 2 — Traffic & errors: Time series for request rate, error rate, and latency.
  • Row 3 — Resources: Gauges for CPU, memory, disk and alerts for nearing capacity.
  • Row 4 — Details & drill-downs: Tables and per-instance panels for root-cause analysis.
This layout supports a natural investigative flow: glance at the top for global health, then move down to isolate affected services and examine resource or dependency issues.
An infographic titled "Building Effective Dashboards" showing four colored rows that outline dashboard sections (Row 1: Status Overview; Row 2: Traffic & Errors; Row 3: Resources; Row 4: Details) with brief panel suggestions. To the right are design tips like top-down information flow, one dashboard per concern, use template variables, set meaningful thresholds, and link dashboards together.
Additional design tips
  • One dashboard per concern (e.g., “API Platform”, “Payments”, “Cluster Health”) to avoid mixed signals.
  • Use template variables for filtering by service, namespace, or region so a single dashboard can serve many contexts.
  • Set meaningful thresholds and color coding for quick interpretation (green/yellow/red).
  • Add links between dashboards: overview → service → instance-level dashboards for rapid navigation.
  • Use annotations for deployments or incidents so trends can be correlated with events.
Bringing it together Grafana is a visualization layer: it queries Prometheus (it does not store metrics). The recommended workflow:
  1. Configure Prometheus as a Grafana data source (first step).
  2. Use PromQL or recording rules to produce reliable metrics.
  3. Pick the panel type that answers each operational question.
  4. Structure dashboards top-down for fast triage and drill-down.
A slide titled "Key Takeaways" with four numbered points summarizing Grafana/Prometheus usage: Grafana as the visualization layer, datasource setup, matching panel types to questions, and structuring dashboards top‑down. The design features a blue gradient left panel and colorful numbered markers down the center.
Links and references Use these resources to automate Grafana provisioning, create recording rules in Prometheus, and adopt consistent dashboard patterns across your organization.

Watch Video