> ## Documentation Index > Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt > Use this file to discover all available pages before exploring further. # Horizontal Sharding > Guide to horizontally sharding Prometheus scrape jobs using hashmod relabeling to distribute targets across multiple instances. This lesson shows how to horizontally shard a heavy Prometheus scrape job so that no single Prometheus instance becomes a bottleneck. We use Prometheus `relabel_configs` with the `hashmod` action to split a single job's targets across multiple Prometheus servers. Why shard? * Large scrape jobs with many targets can increase CPU, memory, and scrape latency for a single Prometheus server. * Sharding distributes scraping load across N Prometheus instances while keeping a single shared target list. Key terms * hashmod: a relabel action that computes `hash(target) % modulus` and stores the result in a label. * shard index: the integer (0..N-1) assigned to a Prometheus instance that determines which subset of targets it keeps. * `__address__`: the label containing the target address (host:port). Overview 1. Compute a hashmod of each target address and store it in a temporary label (commonly `__tmp_hashmod`). 2. Keep only targets whose hashmod equals the shard index assigned to that Prometheus instance. 3. Repeat the same scrape configuration on each instance, changing only the `keep` `regex` to the instance’s shard index. Example: initial single-instance scrape configuration ```yaml theme={null} scrape_configs: - job_name: "prometheus" static_configs: - targets: ["localhost:9090"] labels: app: "prometheus" - job_name: "node" static_configs: - targets: - "192.168.64.8:9100" - "192.168.64.8:9101" - "192.168.64.10:9100" - "192.168.64.10:9101" ``` This single Prometheus instance scrapes all node targets. To shard across multiple Prometheus servers, use `relabel_configs` with `hashmod`. Full common configuration (applies to all Prometheus instances) ```yaml theme={null} global: scrape_interval: 15s evaluation_interval: 15s alerting: alertmanagers: - static_configs: - targets: ["alertmanager:9093"] rule_files: - "rules.yml" scrape_configs: - job_name: "prometheus" static_configs: - targets: ["localhost:9090"] labels: app: "prometheus" - job_name: "node" static_configs: - targets: - "192.168.64.8:9100" - "192.168.64.8:9101" - "192.168.64.10:9100" - "192.168.64.10:9101" ``` Sharding configuration details * Use `modulus: N` where `N` is the number of Prometheus instances (shards). * On each Prometheus instance, set the `keep` `regex` to the instance’s shard index (`0`..`N-1`). * The first relabel writes the hashmod result into `__tmp_hashmod`. The second relabel filters targets based on that value. Prometheus instance A (shard 0) ```yaml theme={null} scrape_configs: - job_name: "node" static_configs: - targets: - "192.168.64.8:9100" - "192.168.64.8:9101" - "192.168.64.10:9100" - "192.168.64.10:9101" relabel_configs: # Step 1: hash the __address__ and write result into a temporary label - source_labels: [__address__] action: hashmod modulus: 2 target_label: __tmp_hashmod # Step 2: keep only targets that map to shard 0 - source_labels: [__tmp_hashmod] regex: "0" action: keep ``` Prometheus instance B (shard 1) ```yaml theme={null} scrape_configs: - job_name: "node" static_configs: - targets: - "192.168.64.8:9100" - "192.168.64.8:9101" - "192.168.64.10:9100" - "192.168.64.10:9101" relabel_configs: - source_labels: [__address__] action: hashmod modulus: 2 target_label: __tmp_hashmod - source_labels: [__tmp_hashmod] regex: "1" action: keep ``` How it works (step-by-step) | Step | Action | Result | | ---- | ------------------------------------------------------------------------------------- | ------------------------------------------------------------------------ | | 1 | Compute `hashmod(__address__) % modulus` via `hashmod` relabel action | Result (`0..N-1`) stored in `__tmp_hashmod` | | 2 | Use `keep` relabel with `source_labels: [__tmp_hashmod]` and `regex: ""` | Only targets with matching hashmod are kept for that Prometheus instance | | 3 | Repeat on each instance with `regex` set to its shard index | Targets are partitioned across the N Prometheus servers | Best practices and considerations * Set `modulus` equal to the number of Prometheus instances. For example, use `modulus: 4` for four shards. * Hash-based distribution is near-uniform with many targets. If you have few targets, distribution may be uneven. * Keep the static target list identical on all Prometheus servers; only the relabel `keep` regex differs. * If you add or remove Prometheus instances (change `modulus`), hash assignments will change; expect some target movement between instances. Verification * Confirm shard behavior using Prometheus’s Status → Targets (Target Health) page for each instance. This page shows which targets each Prometheus server is scraping. * Use the Prometheus UI and logs to verify that each instance scrapes only the expected subset of targets. Links and references * [Prometheus relabeling documentation](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config) * [Prometheus Status → Targets](https://prometheus.io/docs/operating/accessing/) * [Prometheus best practices for scaling](https://prometheus.io/docs/introduction/overview/) The image shows a Prometheus monitoring dashboard displaying target endpoints and their status, with all listed endpoints currently marked as "UP." When choosing the `modulus`, make it equal to the number of Prometheus servers you plan to use. For large numbers of targets, `hashmod` yields an approximately even distribution; for a small number of targets the distribution may be uneven.