Skip to main content
In this lesson we’ll cover service discovery in Prometheus: what it is, why it matters, and the common built-in mechanisms you can use to avoid manually updating your scrape target list. Why is service discovery necessary? Prometheus uses a configuration file (typically prometheus.yml) to define the targets it scrapes. With the static approach, every new server or instance must be added to that config file manually. For example:
Removing decommissioned servers requires remembering to edit prometheus.yml. For a handful of hosts this is manageable, but in dynamic environments—where instances scale up/down or are replaced frequently—manually maintaining prometheus.yml becomes impractical. Service discovery solves this by automatically providing Prometheus with the set of endpoints to scrape. Service discovery allows Prometheus to populate a list of endpoints to scrape
A dark-themed slide explaining service discovery for Prometheus, stating it populates a list of endpoints to scrape that can be dynamically updated as endpoints are created and destroyed. The words "endpoints" and "dynamically" are highlighted and a search box at the top reads "what is service discovery?".
that can be dynamically updated as new endpoints are created and destroyed. Prometheus includes several built-in service discovery integrations so it can discover targets from common platforms and registries rather than relying solely on static_configs. Typical integrations include cloud providers, orchestration systems, and service registries—examples below focus on the environments where Prometheus commonly runs:
A slide titled "Service Discovery" that says "Prometheus has built-in support for several service discovery mechanisms." It shows icons and labels for EC2, Azure, GCE, Consul, Nomad, and Kubernetes.
The original (static) service discovery mechanism—static_configs—is simple but requires explicit listing of every endpoint and manual updates whenever infrastructure changes.
static_configs works well for small, fixed fleets. For dynamic, auto-scaling, or cloud-native environments, prefer one of Prometheus’s dynamic service discovery integrations (for example, Kubernetes, EC2, GCE, Consul, or Nomad) or connect Prometheus to a service registry.
This article reviews the dynamic discovery options Prometheus supports, when to use each, and pointers for configuring them.

​
Why use service discovery?

  • Eliminates manual churn of prometheus.yml in dynamic environments.
  • Keeps scrape target lists accurate as instances are added, removed, or replaced.
  • Integrates with labels and metadata from the environment to make grouping and relabeling easier.
  • Enables automated, infrastructure-driven observability in cloud and container platforms.

​
Common service discovery mechanisms

Use the following table as a quick reference to built-in Prometheus discovery integrations and typical use cases.

​
How Prometheus discovery works (high-level)

  1. Prometheus uses a discovery mechanism (e.g., Kubernetes API, EC2 API, Consul catalog) to obtain a list of scrape targets and their metadata.
  2. Each discovered target is exposed to Prometheus with a set of labels derived from the provider metadata (e.g., instance ID, job, region, service name).
  3. relabel_configs can be applied to transform or filter targets before scraping.
  4. Prometheus continuously refreshes the discovery information so the target list stays up to date as infrastructure changes.

​
Example concepts to keep in mind

  • Use provider labels to group targets (for example, namespace, pod, job in Kubernetes).
  • Use relabel_configs to drop unwanted targets, rewrite labels, or add unified labels across environments.
  • Consider service-level endpoints (e.g., Kubernetes Service objects or Consul services) rather than individual pods/instances when appropriate.

​
Configuration pointers

  • Kubernetes: configure Prometheus to talk to the cluster API and use the kubernetes_sd_configs block in scrape_configs.
  • Cloud providers (EC2/GCE/Azure): configure cloud-specific SD blocks and ensure Prometheus has API access/credentials.
  • Consul: use consul_sd_configs to pull services and optionally filter by tags or use health checks.
For each provider, consult the official Prometheus documentation for the exact config fields and authentication setup:
When using cloud or API-based discovery, ensure Prometheus has the required permissions and network access. Misconfigured credentials or restricted network policies are common causes of discovery failures.

​
Next steps

  • If you run Prometheus in Kubernetes, start with kubernetes_sd_configs and learn how relabel_configs maps Pod/Service metadata to Prometheus labels.
  • For cloud VMs, configure the appropriate cloud provider SD block and secure API credentials using secrets or environment-based IAM roles.
  • Test discovery by enabling verbose Prometheus logs or examining the Targets page (/targets) in the Prometheus UI to verify discovered endpoints and label sets.

Watch Video