Fix a Hot Shard

In this lesson we’ll intentionally direct follower traffic to a shard cluster, observe a single shard become overloaded (a “hot shard”), and then apply operational fixes to rebalance traffic and restore even load across shards.

This hands-on exercise shows how to detect a hot-shard condition and apply targeted remediation. You’ll practice traffic routing, metric inspection, and corrective actions such as adjusting routing weights, scaling, or reassigning shard ranges.

What you’ll accomplish

Route follower traffic to the shard cluster to reproduce a hot-shard scenario.
Monitor shard metrics (request rate, latency, CPU/memory) to identify the hot shard.
Apply fixes to redistribute traffic and reduce overload on the hot shard.
Validate that the cluster returns to balanced traffic distribution.

Prerequisites

Requirement	Example / How to check
Access to shard cluster and router control plane	`kubectl` access or control-plane API credentials
Metrics/monitoring visible (Prometheus/Grafana)	`http_requests_total`, `container_cpu_usage_seconds_total`
Permission to modify routing or shard configuration	Ability to update router weights or service endpoints
Load generator available for testing	`fortio`, `wrk`, or a simple `curl` loop

Step-by-step guide

Route follower traffic to the shard cluster

Purpose: Direct follower requests so they hit the shard layer. This lets you reproduce imbalanced traffic patterns.
How to do it:
- Update your router / gateway to send follower traffic to the shard cluster. This is typically a configuration change in your control plane or ingress (e.g., altering weights, endpoint lists, or route rules).
- Example (conceptual placeholder — adapt to your control plane):
  - Update route weight: POST /api/routes/<route-id>/weights with payload to set shard-cluster weight to 1.0.
- If you use Kubernetes Services and want to target a specific shard for testing, update the Endpoints object or point a test client directly to http://<shard-ip>:<port>.

Generate load to reproduce the hot shard

Purpose: Create traffic to cause one shard to become overloaded so you can observe symptoms.

Example commands:

Using a simple loop:

for i in {1..1000}; do curl -s -o /dev/null -w "%{http_code}\n" http://<follower-endpoint>/followers; done

Using fortio (recommended for realistic load):

fortio load -c 50 -qps 200 -t 60s http://<follower-endpoint>/followers

Replace <follower-endpoint> with your test endpoint.

Observe and identify the hot shard

What to watch (Prometheus/Grafana queries):

Request rate by shard:

rate(http_requests_total{job="shard-service"}[1m]) by (shard)

95th percentile latency by shard:

histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{job="shard-service"}[5m])) by (le, shard))

CPU usage (per pod):

sum(rate(container_cpu_usage_seconds_total{namespace="shards"}[5m])) by (pod)

Quick kubectl checks:

List pods and status:
kubectl get pods -n shards -o wide
Resource usage (metrics-server):
kubectl top pods -n shards

Symptom mapping:

Symptom	Likely diagnostic	Quick check
High request rate and latency on one shard	Shard receiving disproportionate traffic	Prometheus request-rate by `shard`
CPU or memory saturation on one shard	Insufficient capacity or bad routing	`kubectl top pods` and pod restarts
Errors (5xx) concentrated on one shard	Overload causing timeouts or dropped requests	`rate(http_requests_total{code=~"5.."}[1m]) by (shard)`

You will make live changes to routing or shard assignments. Validate changes in a staging environment first if possible. Roll back quickly if error rates increase or user impact grows.

Apply fixes to rebalance traffic Choose one or more actions depending on diagnostics:

Adjust routing weights (preferred quick fix)
- Reduce weight to the hot shard and increase weight to other shards.
- Example conceptual API call:
  # Pseudocode — adapt to your routing control plane curl -X PATCH https://router.example.com/routes/<route-id> \ -d '{"shardWeights": {"shard-1": 0.2, "shard-2": 0.4, "shard-3": 0.4}}'
- Verify the new distribution quickly using request-rate metrics.

Scale the overloaded shard

If the shard is CPU-bound, scale replicas or increase resources:

kubectl scale deployment shard-2 -n shards --replicas=4
kubectl set resources deployment shard-2 -n shards --limits=cpu=2000m,memory=2Gi

Reassign shard ranges (reshard)
- If the root cause is uneven key distribution, move some key ranges from the hot shard to others.
- Reshard steps typically include:
  - Pause writes to affected ranges (if your system requires it).
  - Move key ranges or update shard metadata in the control plane.
  - Drain and re-route traffic for moved ranges.
- Exact commands depend on your sharding system — follow your platform’s resharding procedure.
Evict or restart unhealthy instances
- If a pod has degraded performance but is not resource-limited, restart it:
  kubectl rollout restart deployment shard-2 -n shards

Validate remediation

Confirm request rate and latency have evened out:
- Re-run the Prometheus queries from step 3 and confirm no single shard dominates traffic.
Confirm resource usage normalized:
kubectl top pods -n shards

Check error rates dropped:

rate(http_requests_total{job="shard-service",code=~"5.."}[5m]) by (shard)

If you adjusted routing weights, validate routing config is persisted and that the control plane has applied the new weights.

Postmortem checklist (after fix)

Record the root cause (routing config, key distribution, insufficient capacity).
Capture the exact remediation steps and timestamps.
Consider long-term mitigations:
- Improve hashing or shard key selection.
- Add autoscaling rules for shard replicas.
- Implement more granular routing or a circuit breaker for hot shards.

Useful references

Proceed to the hands-on steps in your environment to reproduce a hot shard and practice the fixes described above.

Why One Database Isn't Enough

Sharding Fundamentals

Routing and Hot Shards

Shard Keys and Partitioning

Resharding and Cross-Shard Queries

Watch Video

Practice Lab