Skip to main content
In this lesson we demonstrate Prometheus federation: how to aggregate metrics from multiple Prometheus servers (each responsible for a datacenter) into a single global Prometheus instance for a unified “single pane of glass”. Use case: you run one Prometheus per datacenter to keep scraping load local, but want a global view across datacenters without opening multiple UIs.

​
Scenario overview

  • Prometheus 1 — scrapes nodes in Datacenter 1.
  • Prometheus 2 — scrapes nodes in Datacenter 2.
  • Global Prometheus — federates aggregated metrics from Prometheus 1 and Prometheus 2 using the /federate endpoint.
Example configuration for Datacenter 1 (Prometheus 1)
Example configuration for Datacenter 2 (Prometheus 2)
Both local Prometheus servers scrape only their local nodes. This pattern limits cross-datacenter network load and distributes scraping work. You can verify each Prometheus is scraping its targets (example UI view):
This image shows a Prometheus monitoring interface displaying the status of various endpoints, including nodes and Prometheus instances, with their last scrape times and states indicated as "UP".

​
Problem: multiple UIs and potential overload

If you need to inspect metrics from many datacenters, opening each Prometheus UI is inconvenient. A global Prometheus can scrape the local Prometheus servers’ federation endpoint (/federate) to provide a single view. Initial global Prometheus scrape job for federation (example)
This configuration instructs the global Prometheus to query each local Prometheus at /federate and request timeseries that match job="node". Querying the global Prometheus for a node metric (example UI):
The image shows the Prometheus query interface open in a web browser, with a query being entered for metrics starting with "node". Various metric options and their statuses are displayed below the query box.
Example result returned by the global Prometheus after federation:
This shows timeseries from both datacenters are visible in the global Prometheus. However, federating instance-level metrics like these introduces important design risks.

​
Why federating instance-level metrics is dangerous

  • High cardinality: sending every node/instance metric to a central Prometheus multiplies label combinations and can overwhelm the server.
  • Duplicate metric collisions: if multiple sources expose the same metric name and identical label sets (e.g., job="node"), the global Prometheus may receive samples with the same metric+labels and identical timestamps but different values. Prometheus treats this as conflicting samples and drops them, logging errors.
Example of ingestion error logs (cleaned):
If the same metric name and label set are produced by multiple sources, Prometheus can reject samples with identical timestamps but differing values. This commonly occurs when federating raw instance-level metrics.
Best practice: Do not federate high-cardinality, instance-level metrics to a single global Prometheus. Instead, federate aggregated, lower-cardinality metrics computed locally (recording rules) and include a distinguishing label (for example datacenter="dc1").

​
Solution: aggregate locally with recording rules and add a datacenter label

  1. On each local Prometheus, define recording rules that aggregate instance-level series (sums, rates, etc.).
  2. Add a datacenter label to those recorded metrics to identify their origin.
  3. Configure the global Prometheus to federate only those recorded (aggregated) metric names.
Example recording rules file (rules.yaml) — Datacenter 1
For Datacenter 2, create the same rules but set datacenter: "dc2". Include the rules file in each local Prometheus configuration:
Restart the local Prometheus servers so they evaluate the recording rules and publish the aggregated metrics. Each local Prometheus will now expose recorded metrics such as:
Adjust the global Prometheus federation scrape to request only the recorded metric names:
With this setup:
  • Local Prometheus instances compute aggregated metrics and attach a datacenter label.
  • The global Prometheus federates only aggregated metrics (low cardinality).
  • The datacenter label distinguishes origins and prevents ingestion conflicts.
Example query on the global Prometheus, asking for memory available from datacenter 2:
Example returned value:

​
Quick reference: federation do’s and don’ts

  1. Keep local scraping per datacenter for instance-level metrics.
  2. Create recording rules locally that aggregate instance-level metrics and add a datacenter label.
  3. Federate only the aggregated recorded metrics to the global Prometheus.
  4. Query the global Prometheus for datacenter-level metrics to get a single pane of glass without overwhelming it.

Watch Video