Skip to main content
In this lesson we demonstrate Prometheus federation: how to aggregate high-level metrics from multiple Prometheus servers (one per datacenter) into a single global Prometheus instance. The goal is a single-pane-of-glass for cross-datacenter visibility while avoiding overwhelming the global server with instance-level metrics. Scenario:
  • Prometheus 1 collects metrics for all servers in datacenter 1.
  • Prometheus 2 collects metrics for all servers in datacenter 2.
  • Prometheus-global is the federated (global) Prometheus that queries the other two.
Environment quick check (local machines have Prometheus release directories):

​
Datacenter-level Prometheus configuration

Each datacenter-level Prometheus scrapes node exporters inside its datacenter. This keeps instance-level scraping localized and reduces load on a central server. Example scrape configuration on Prometheus 1 (datacenter 1):
Prometheus 2 (datacenter 2) would have a similar node job targeting the instances in that datacenter. Running one Prometheus per datacenter avoids saturating a single Prometheus server with scrapes from every instance across all datacenters.

​
Why not federate all instance-level metrics?

If you must inspect metrics across datacenters, logging into 10 different Prometheus servers is painful. A global Prometheus that federates selected metrics provides a single view. However, federating raw instance-level metrics from many datacenters causes two problems:
  1. Scale: pulling every instance-level time series into the global server will overwhelm it.
  2. Label collision: identical metric names and identical label sets from different datacenters produce samples with the same metric + label set at the same timestamp. When values differ, Prometheus will drop samples and log warnings.
A naive global scrape of instance-level metrics might look like this:
When running a naive federation you may see logs like:
This happens because the global Prometheus cannot distinguish identical metric+label combinations coming from different datacenters, so it must drop conflicting samples.
Best practice: Federate only aggregated / rollup metrics (datacenter-level summaries), not raw instance-level metrics. This minimizes load and avoids label collisions.

​
Proper approach: compute aggregates locally, federate aggregates

Instead of federating instance-level metrics, compute datacenter-level aggregates via recording rules on each local Prometheus. Then have the global Prometheus federate only those aggregated series. Example recording rules file, rules.yaml, to run on each datacenter Prometheus:
Load the rules on each local Prometheus by referencing the file in prometheus.yml:
Restart each local Prometheus so the recording rules are evaluated. Each local Prometheus will now expose aggregated series such as:
  • node:cpu_seconds:sum_rate5m{job="node"}
  • node:memory_MemAvailable_bytes:sum{job="node"}
These aggregated series represent datacenter-level summaries, not individual instances.

​
Configure global Prometheus to federate aggregates and tag datacenters

To prevent label collisions and to store aggregates from multiple datacenters separately, attach a distinct datacenter label to each federation target. Add the datacenter label to the static_configs entry for each remote Prometheus target — static target labels are appended to all scraped metrics from that target. Example prometheus.yml for the global Prometheus:
Key points:
  • metrics_path: "/federate" requests only selected series from the remote Prometheus.
  • params.match[] lists the aggregated series to fetch; limiting this reduces the global server’s load.
  • Static labels (e.g. datacenter) are added to all metrics scraped from that target, giving you a distinguishing label for each aggregated series.
Be careful with honor_labels. If honor_labels: true and a remote metric already exposes a datacenter label, the remote label will be preserved and may conflict with the target label. Choose a label name that remote metrics won’t use, or set honor_labels: false to let target labels override remote metric labels.
After restarting the local and global Prometheus instances, the global Prometheus should now ingest only the aggregated series — two per datacenter in this example — each tagged with datacenter="dc1" or datacenter="dc2".

​
Example queries on the global Prometheus

Aggregated memory per datacenter:
Example result:
Aggregated CPU per datacenter:

​
Quick comparison

​
Summary / Best practices

  • Do not federate raw instance-level metrics from many datacenters into a single global Prometheus — it will likely overwhelm the global instance and cause label collisions.
  • Compute datacenter-level aggregates with recording rules on each local Prometheus, then federate only those aggregated series.
  • Ensure federated series include a distinguishing label (for example datacenter) so the global server can store series from multiple sources without label collisions.
  • Use params.match[] on the /federate scrape to limit the set of metrics fetched.
  • Assign a datacenter identifier via static target labels (or via relabeling) and ensure it does not clash with existing metric labels; consider honor_labels behavior when deciding whether target labels should override remote metric labels.
This design provides a single-pane-of-glass view across datacenters while minimizing load on the global Prometheus server.

Watch Video