Skip to main content
This guide shows how to implement horizontal sharding for a Prometheus scrape job so you can scale scraping across multiple Prometheus servers. Horizontal sharding distributes a large set of targets among N Prometheus instances so no single server becomes overloaded. Key benefits:
  • Scale scraping horizontally with minimal changes to target discovery.
  • Keep a single canonical job_name and target list while splitting work at scrape time.
  • Works with static targets, file-based service discovery, or many SD integrations.

​
Scenario

  • You have one job, e.g. job_name: "node", with many targets.
  • You want to split that job across N Prometheus servers (example below uses N = 2), so each server scrapes only a subset (shard) of the targets.

​
High-level approach

  1. Configure every Prometheus server with the same static target list for the node job.
  2. Use a relabeling pipeline on each Prometheus instance to compute a numeric shard index from the __address__ label using the hashmod action.
  3. Use a keep relabel action to retain only the targets that match the shard index for that Prometheus instance.
Important: every Prometheus instance uses the same job_name and the same targets list. Relabeling is what causes each instance to keep only its shard of targets.
Labels whose names begin with __ are internal to Prometheus relabeling. A temporary label like __tmp_hashmod will not show up as a metric label after relabeling; it is used only during the relabeling pipeline.

​
Example: base prometheus.yml (shared target list)

All Prometheus instances should share the same job and target list. Below is an example base configuration containing a node job with four targets.

​
Relabeling to split across two Prometheus instances

Summary of how the relabeling works:
  • Compute a hash of the target address and take modulo N to produce a shard index in the range 0..N-1.
  • Keep only targets matching the shard index for the current instance.
Table: Relabel parameters

​
Prometheus instance A (shard index = 0)

Add relabel_configs to compute __tmp_hashmod from __address__ and keep only targets where the computed value equals 0.

​
Prometheus instance B (shard index = 1)

Same relabel pipeline, but keep the shard index 1.
Notes:
  • To shard across N Prometheus servers, use modulus: N and set the regex in the keep action to the shard index for each server (values 0..N-1).
  • The hashing function provides an approximately even distribution for many targets; with very few targets you may see uneven shard sizes.
  • Make sure all servers use the identical target list; differing target lists will result in differing shard membership.

​
Verifying shards in the Prometheus UI

  • On each Prometheus instance go to Status → Targets (Target Health).
  • Each instance will list only the targets assigned to that instance’s shard index.
Example Target output (Prometheus instance A)
Example Target output (Prometheus instance B)
When you horizontally shard scraping, be aware that rule evaluation and alerting may be affected:
  • Recording rules and alerts evaluated on each Prometheus will only see the metrics scraped by that instance.
  • If you use Alertmanager, ensure deduplication and routing are configured appropriately to avoid duplicate or missing alerts.
Consider using long-term storage, a centralized evaluation pipeline, or a global rules evaluation layer if you need consolidated views, globally-accurate recording rules, or deduplicated alerts.

​
When to use horizontal sharding

  • Large fleets where a single Prometheus cannot handle scrape load or target churn.
  • Environments where you prefer stateless, independent Prometheus instances that each handle a subset of targets.
  • As a simple scaling strategy before moving to remote-write or a centralized long-term storage solution.

​
Troubleshooting tips

  • If some instances show fewer targets than expected, verify every instance is using the exact same target list (including label differences).
  • If shard distribution appears skewed, consider increasing the number of targets or using a different sharding modulus; remember hashing will even out with larger target counts.
  • Use relabel_configs logs or dry-run relabel simulation tools when available to validate label transformations.

​
Summary

  • Horizontal sharding with relabel_configs and hashmod is an effective way to split scraping across multiple Prometheus instances.
  • Keep the same target list on each Prometheus instance and use hashmod + keep to select the shard.
  • Adjust modulus and the regex in the keep action according to the number of shards and the shard index for each Prometheus server.

Watch Video