> ## Documentation Index
> Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Fix a Hot Shard
> Guide to reproduce and fix an overloaded shard by routing traffic, monitoring metrics, and applying fixes such as weight changes, scaling, or resharding.
In this lesson we’ll intentionally direct follower traffic to a shard cluster, observe a single shard become overloaded (a "hot shard"), and then apply operational fixes to rebalance traffic and restore even load across shards.
This hands-on exercise shows how to detect a hot-shard condition and apply targeted remediation. You’ll practice traffic routing, metric inspection, and corrective actions such as adjusting routing weights, scaling, or reassigning shard ranges.
What you'll accomplish
* Route follower traffic to the shard cluster to reproduce a hot-shard scenario.
* Monitor shard metrics (request rate, latency, CPU/memory) to identify the hot shard.
* Apply fixes to redistribute traffic and reduce overload on the hot shard.
* Validate that the cluster returns to balanced traffic distribution.
Prerequisites
| Requirement | Example / How to check |
| - | - |
| Access to shard cluster and router control plane | `kubectl` access or control-plane API credentials |
| Metrics/monitoring visible (Prometheus/Grafana) | `http_requests_total`, `container_cpu_usage_seconds_total` |
| Permission to modify routing or shard configuration | Ability to update router weights or service endpoints |
| Load generator available for testing | `fortio`, `wrk`, or a simple `curl` loop |
Step-by-step guide
1. Route follower traffic to the shard cluster
* Purpose: Direct follower requests so they hit the shard layer. This lets you reproduce imbalanced traffic patterns.
* How to do it:
* Update your router / gateway to send follower traffic to the shard cluster. This is typically a configuration change in your control plane or ingress (e.g., altering weights, endpoint lists, or route rules).
* Example (conceptual placeholder — adapt to your control plane):
* Update route weight: `POST /api/routes//weights` with payload to set `shard-cluster` weight to `1.0`.
* If you use Kubernetes Services and want to target a specific shard for testing, update the Endpoints object or point a test client directly to `http://:`.
2. Generate load to reproduce the hot shard
* Purpose: Create traffic to cause one shard to become overloaded so you can observe symptoms.
* Example commands:
* Using a simple loop:
```bash theme={null}
for i in {1..1000}; do curl -s -o /dev/null -w "%{http_code}\n" http:///followers; done
```
* Using `fortio` (recommended for realistic load):
```bash theme={null}
fortio load -c 50 -qps 200 -t 60s http:///followers
```
* Replace `` with your test endpoint.
3. Observe and identify the hot shard
* What to watch (Prometheus/Grafana queries):
* Request rate by shard:
```PromQL theme={null}
rate(http_requests_total{job="shard-service"}[1m]) by (shard)
```
* 95th percentile latency by shard:
```promql theme={null}
histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{job="shard-service"}[5m])) by (le, shard))
```
* CPU usage (per pod):
```PromQL theme={null}
sum(rate(container_cpu_usage_seconds_total{namespace="shards"}[5m])) by (pod)
```
* Quick `kubectl` checks:
* List pods and status:
```bash theme={null}
kubectl get pods -n shards -o wide
```
* Resource usage (metrics-server):
```bash theme={null}
kubectl top pods -n shards
```
* Symptom mapping:
| Symptom | Likely diagnostic | Quick check |
| - | - | - |
| High request rate and latency on one shard | Shard receiving disproportionate traffic | Prometheus request-rate by `shard` |
| CPU or memory saturation on one shard | Insufficient capacity or bad routing | `kubectl top pods` and pod restarts |
| Errors (5xx) concentrated on one shard | Overload causing timeouts or dropped requests | `rate(http_requests_total{code=~"5.."}[1m]) by (shard)` |
You will make live changes to routing or shard assignments. Validate changes in a staging environment first if possible. Roll back quickly if error rates increase or user impact grows.
4. Apply fixes to rebalance traffic
Choose one or more actions depending on diagnostics:
* Adjust routing weights (preferred quick fix)
* Reduce weight to the hot shard and increase weight to other shards.
* Example conceptual API call:
```bash theme={null}
# Pseudocode — adapt to your routing control plane
curl -X PATCH https://router.example.com/routes/ \
-d '{"shardWeights": {"shard-1": 0.2, "shard-2": 0.4, "shard-3": 0.4}}'
```
* Verify the new distribution quickly using request-rate metrics.
* Scale the overloaded shard
* If the shard is CPU-bound, scale replicas or increase resources:
```bash theme={null}
kubectl scale deployment shard-2 -n shards --replicas=4
kubectl set resources deployment shard-2 -n shards --limits=cpu=2000m,memory=2Gi
```
* Reassign shard ranges (reshard)
* If the root cause is uneven key distribution, move some key ranges from the hot shard to others.
* Reshard steps typically include:
* Pause writes to affected ranges (if your system requires it).
* Move key ranges or update shard metadata in the control plane.
* Drain and re-route traffic for moved ranges.
* Exact commands depend on your sharding system — follow your platform's resharding procedure.
* Evict or restart unhealthy instances
* If a pod has degraded performance but is not resource-limited, restart it:
```bash theme={null}
kubectl rollout restart deployment shard-2 -n shards
```
5. Validate remediation
* Confirm request rate and latency have evened out:
* Re-run the Prometheus queries from step 3 and confirm no single shard dominates traffic.
* Confirm resource usage normalized:
```bash theme={null}
kubectl top pods -n shards
```
* Check error rates dropped:
```PromQL theme={null}
rate(http_requests_total{job="shard-service",code=~"5.."}[5m]) by (shard)
```
* If you adjusted routing weights, validate routing config is persisted and that the control plane has applied the new weights.
Postmortem checklist (after fix)
* Record the root cause (routing config, key distribution, insufficient capacity).
* Capture the exact remediation steps and timestamps.
* Consider long-term mitigations:
* Improve hashing or shard key selection.
* Add autoscaling rules for shard replicas.
* Implement more granular routing or a circuit breaker for hot shards.
Useful references
* [Kubernetes Documentation](https://kubernetes.io/docs/)
* [Prometheus Querying Basics](https://prometheus.io/docs/prometheus/latest/querying/basics/)
* [Grafana Dashboards](https://grafana.com/docs/grafana/latest/dashboards/)
* [Service Mesh / Router docs (Envoy/Traefik/NGINX)](https://www.envoyproxy.io/)
Proceed to the hands-on steps in your environment to reproduce a hot shard and practice the fixes described above.
This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.