Skip to main content
This lesson shows how to horizontally shard a heavy Prometheus scrape job so that no single Prometheus instance becomes a bottleneck. We use Prometheus relabel_configs with the hashmod action to split a single job’s targets across multiple Prometheus servers. Why shard?
  • Large scrape jobs with many targets can increase CPU, memory, and scrape latency for a single Prometheus server.
  • Sharding distributes scraping load across N Prometheus instances while keeping a single shared target list.
Key terms
  • hashmod: a relabel action that computes hash(target) % modulus and stores the result in a label.
  • shard index: the integer (0..N-1) assigned to a Prometheus instance that determines which subset of targets it keeps.
  • __address__: the label containing the target address (host:port).
Overview
  1. Compute a hashmod of each target address and store it in a temporary label (commonly __tmp_hashmod).
  2. Keep only targets whose hashmod equals the shard index assigned to that Prometheus instance.
  3. Repeat the same scrape configuration on each instance, changing only the keep regex to the instance’s shard index.
Example: initial single-instance scrape configuration
This single Prometheus instance scrapes all node targets. To shard across multiple Prometheus servers, use relabel_configs with hashmod. Full common configuration (applies to all Prometheus instances)
Sharding configuration details
  • Use modulus: N where N is the number of Prometheus instances (shards).
  • On each Prometheus instance, set the keep regex to the instance’s shard index (0..N-1).
  • The first relabel writes the hashmod result into __tmp_hashmod. The second relabel filters targets based on that value.
Prometheus instance A (shard 0)
Prometheus instance B (shard 1)
How it works (step-by-step) Best practices and considerations
  • Set modulus equal to the number of Prometheus instances. For example, use modulus: 4 for four shards.
  • Hash-based distribution is near-uniform with many targets. If you have few targets, distribution may be uneven.
  • Keep the static target list identical on all Prometheus servers; only the relabel keep regex differs.
  • If you add or remove Prometheus instances (change modulus), hash assignments will change; expect some target movement between instances.
Verification
  • Confirm shard behavior using Prometheus’s Status → Targets (Target Health) page for each instance. This page shows which targets each Prometheus server is scraping.
  • Use the Prometheus UI and logs to verify that each instance scrapes only the expected subset of targets.
Links and references
The image shows a Prometheus monitoring dashboard displaying target endpoints and their status, with all listed endpoints currently marked as "UP."
When choosing the modulus, make it equal to the number of Prometheus servers you plan to use. For large numbers of targets, hashmod yields an approximately even distribution; for a small number of targets the distribution may be uneven.

Watch Video