Scaling Prometheus

In this lesson we cover how to scale Prometheus for capacity and high availability (HA). Prometheus is built to scale, but a single server can hit limits as you add more scrape jobs, targets, or increase metric cardinality. When scrape volume grows, one server may struggle to collect and store all time series; configuration reloads can also cause staleness until scrapes resume. This guide explains practical approaches:

Vertical scaling (team- or service-based isolation)
Horizontal scaling (sharding a scrape job across multiple Prometheus servers)
Federation and aggregation for a global view
Alternatives (Thanos, Cortex) for unified querying and long-term storage

Vertical scaling

Vertical scaling assigns ownership: each Prometheus instance monitors a distinct set of services or applications. This limits time series cardinality per instance and lets teams manage their own alerting and recording rules independently.

Benefits of vertical scaling:

Clear ownership and accountability for metric cardinality and configuration.
Teams can experiment with alerting and dashboards without impacting others.
Limits blast radius when a single Prometheus instance misbehaves.

Horizontal scaling (sharding)

Horizontal sharding splits a large scrape job across multiple Prometheus servers. This is ideal when one job has many instances and a single Prometheus cannot scrape them all in time.

A diagram titled "Horizontal Scaling" showing two Prometheus service shards each collecting from multiple instances of "Service A" running on four machines labeled Machine01–Machine04. Red rounded boxes represent the Prometheus shards and service instances, and blue boxes represent the machines.

Sharding example (2 shards)

Configure the full target list on every Prometheus server.
Use relabel_configs with hashmod to deterministically keep a portion of targets on each server.

Prometheus server 1 (keeps hash result 0):

- job_name: "node"
  scheme: http
  static_configs:
    - targets: ["192.168.64.8:9100", "192.168.64.10:9100"]
  relabel_configs:
    - source_labels: ["__address__"]
      action: hashmod
      modulus: 2
      target_label: "__tmp_hashmod"
    - action: keep
      source_labels: ["__tmp_hashmod"]
      regex: "0"

Prometheus server 2 (keeps hash result 1):

- job_name: "node"
  scheme: http
  static_configs:
    - targets: ["192.168.64.8:9100", "192.168.64.10:9100"]
  relabel_configs:
    - source_labels: ["__address__"]
      action: hashmod
      modulus: 2
      target_label: "__tmp_hashmod"
    - action: keep
      source_labels: ["__tmp_hashmod"]
      regex: "1"

Key sharding notes:

Set modulus equal to the number of shards (e.g., modulus: 5 for five servers, matching regex 0 through 4 across servers).
Keep the same full target list on all servers; relabeling determines which server retains each target.
Distribution is deterministic but not guaranteed perfectly even — monitor and rebalance if necessary.

Other tuning options for a single server

If you want to avoid multiple Prometheus instances, tune a single server to reduce load.

Increase scrape intervals for slow-changing metrics:

scrape_configs:
  - job_name: 'slow-metrics-job'
    scrape_interval: 1m
    static_configs:
      - targets: ['localhost:9100']

Limit samples per scrape to prevent a single target from producing too many time series:

scrape_configs:
  - job_name: 'example'
    static_configs:
      - targets: ['localhost:8080']
    sample_limit: 10000    # limit time series per scrape to 10,000

Federation: global view without copying everything

Federation is commonly used to aggregate metrics from multiple Prometheus servers (for example, one per datacenter). A global Prometheus can scrape the /federate endpoint of per-datacenter Prometheus servers to provide a single-pane-of-glass view.

A diagram titled "Prometheus Federation" showing a Global Prometheus node at the top connected by dashed arrows to three Prometheus instances running on separate blue data server boxes below. A red "Limit" bar is shown above the global node.

Basic federation job (pull all metrics from job="node"):

scrape_configs:
  - job_name: "prometheus-federation"
    honor_labels: true
    metrics_path: "/federate"
    params:
      match[]:
        - '{job="node"}'
    static_configs:
      - targets: ["192.168.64.7:9090", "192.168.64.11:9090"]

Use metrics_path: "/federate" to scrape another Prometheus’ federate endpoint.
params.match[] controls which series the federated endpoint returns.
honor_labels: true preserves labels from the source Prometheus.

Important: avoid copying entire instance-level metrics into the global Prometheus — doing so defeats distributed load and will likely overwhelm the global server.

Aggregate and record rules before federating

A best practice is to calculate aggregated series (via recording rules) on each per-datacenter Prometheus, then federate only those aggregated metrics to the global server. Recording rules example (on each datacenter Prometheus):

groups:
  - name: aggregated-node-metrics
    rules:
      # CPU usage by job and datacenter (5m rate summed)
      - record: node:cpu_seconds:sum_rate5m
        expr: sum(rate(node_cpu_seconds_total[5m])) by (job, datacenter)
      # Memory available by job and datacenter
      - record: node:memory_MemAvailable_bytes:sum
        expr: sum(node_memory_MemAvailable_bytes) by (job, datacenter)

Global Prometheus federating only recorded metrics:

scrape_configs:
  - job_name: "prometheus-federation"
    honor_labels: true
    metrics_path: "/federate"
    params:
      match[]:
        - 'node:cpu_seconds:sum_rate5m'
        - 'node:memory_MemAvailable_bytes:sum'
    static_configs:
      - targets: ["192.168.64.7:9090", "192.168.64.11:9090"]

Avoid duplicate series — label your data sources

If multiple datacenter Prometheus servers expose identical metric names and labels, the global Prometheus will see them as duplicate series. Make series unique by adding a datacenter label. Add a relabeling step on the global Prometheus or on the per-datacenter Prometheus:

relabel_configs:
  - target_label: datacenter
    replacement: dc1

On the other datacenter, set replacement: dc2.

Ensure recording rules group by datacenter so you can query metrics per datacenter:

node:cpu_seconds:sum_rate5m{datacenter="dc2"}

Do not use federation to copy entire Prometheus instances or to proxy queries. Federation is intended for pulling specific aggregated metrics, not for duplicating instance-level series or acting as a proxy for full query traffic.

Federation summary

Alternatives: Thanos and global querying

If you want a unified query layer and long-term storage without restricting to aggregated metrics, consider solutions such as Thanos or Cortex:

Thanos (https://thanos.io/) — attaches sidecars to Prometheus instances and provides global query, downsampling, and object-store-backed long-term storage.
Cortex (https://cortexmetrics.io/) — scalable, multi-tenant Prometheus-compatible system offering a global view with long-term storage.

These systems let you query across many Prometheus instances without copying all instance-level series into a single Prometheus server.

A diagram titled "Querying Across Multiple Prometheus Servers – Thanos" showing Thanos in the center with dashed lines down to three Prometheus/data server boxes. A green user icon on the right connects to Thanos, indicating centralized querying across those servers.

Comparison at a glance

Approach	Best for	Pros	Cons
Vertical scaling	Team isolation, ownership	Simple, isolates cardinality and experiments	Multiple Prometheus instances to operate
Horizontal sharding	Very large scrape jobs	Distributes scraping load deterministically	Requires consistent target lists and relabeling logic
Federation (aggregated metrics)	Global summaries across datacenters	Lightweight global Prometheus, only aggregated series	Not suitable for full instance-level queries
Thanos / Cortex	Unified querying and long-term storage	Global queries, scaling, retention	Additional components and operational complexity

Summary

Use vertical scaling to grant teams isolated Prometheus instances and ownership.
Use horizontal sharding to split large scrape jobs across multiple Prometheus servers.
Tune scrape intervals and sample_limit for single-server protection.
Use federation only for carefully selected aggregated metrics, and add labels (e.g., datacenter) to avoid duplicates.
For unified, instance-level querying and long-term storage consider Thanos or Cortex instead of copying instance-level series via federation.

Tip: Monitor Prometheus runtime metrics (e.g., prometheus_tsdb_head_series, prometheus_engine_query_duration_seconds) to detect cardinality or query-load issues early and choose the appropriate scaling strategy.

Introduction

Observability Fundamentals

Prometheus Fundamentals

PromQL

Dashboarding & Visualization

Application Instrumentation

Service Discovery

Push Gateway

Alerting

Monitoring Kubernetes

Conclusion

Scaling & Long Term Storage

Scaling Prometheus

Vertical scaling

Horizontal scaling (sharding)

Other tuning options for a single server

Federation: global view without copying everything

Aggregate and record rules before federating

Avoid duplicate series — label your data sources

Federation summary

Alternatives: Thanos and global querying

Comparison at a glance

Further reading and references

Summary

Watch Video

​Vertical scaling

​Horizontal scaling (sharding)

​Other tuning options for a single server

​Federation: global view without copying everything

​Aggregate and record rules before federating

​Avoid duplicate series — label your data sources

​Federation summary

​Alternatives: Thanos and global querying

​Comparison at a glance

​Further reading and references

​Summary

Watch Video

Vertical scaling

Horizontal scaling (sharding)

Other tuning options for a single server

Federation: global view without copying everything

Aggregate and record rules before federating

Avoid duplicate series — label your data sources

Federation summary

Alternatives: Thanos and global querying

Comparison at a glance

Further reading and references

Summary