Skip to main content
This lesson explains a clear, easy-to-visualize resharding strategy called the fixed-buckets approach. It is an alternative to consistent hashing that trades a tiny bit of indirection (user → bucket → shard) for predictable resharding behavior and much less data movement when you add or remove shards. How it works at a high level:
  • Choose a fixed number of buckets (for example, 1000 buckets, numbered 0–999).
  • Hash each user ID into one of those buckets using a deterministic hash.
  • Maintain a small bucket-to-shard map that assigns each bucket to one shard.
Routing is a two-step lookup:
  1. Compute the bucket from the user ID: bucket = hash(user_id) % 1000.
  2. Look up the shard for that bucket in the bucket map, then route the request to that shard.
So, for example, Chris might hash to bucket 371 and Leo to bucket 120. Their bucket assignments never change because the number of buckets and the modulo operation remain fixed. What can change is the bucket-to-shard mapping.
A stylized diagram of hash buckets and sharding showing five columns of 200 small squares above cylinders labeled Shard 0–3 and New Shard, with some buckets highlighted and annotated (e.g., "Leo 120", "Chris 371"). A "Bucket Map" table at the right lists bucket-to-shard assignments.
Key invariant
  • A user always stays in the same bucket; only the bucket-to-shard mapping can change.
  • Because the number of buckets is fixed, this strategy is commonly called the fixed-buckets approach.
Benefits and an example
  • If you have 4 shards and 1000 buckets, each shard initially owns roughly 1000 / 4 = 250 buckets.
  • Adding a 5th shard means redistributing buckets so each shard owns about 1000 / 5 = 200 buckets. Each existing shard hands off about 50 buckets to the new shard, so only ~200 out of 1000 buckets move (≈20% of the buckets), rather than moving every user.
Table: Basic parameters and what changes during resharding Practical routing example Moving a bucket (safe migration) Suppose we need to move bucket 602 from shard 2 to a new shard. Rather than moving users individually, we move the whole bucket. To keep the system available and consistent, use this four-step procedure:
  1. Copy: In the background, copy all existing rows for bucket 602 from shard 2 to the new shard. Shard 2 continues to serve traffic for the bucket while this bulk copy runs.
A dark-themed system diagram showing a copy operation of "bucket 602" from one shard to a new shard. It includes user icons, a bucket map, and arrows between two cylindrical storage containers labeled "Shard 2" and "New Shard."
  1. Catch up (replicate incremental changes): While the bulk copy is running, stream or replay new writes for bucket 602 from shard 2 to the new shard so the new copy can catch up with ongoing updates.
  2. Cutover (brief pause): Pause writes for bucket 602 for a very short window (often a second or a few seconds), apply any final changes to the new shard so both copies are identical, and atomically update the bucket map to point bucket 602 at the new shard. After this update, reads and writes for bucket 602 are routed to the new shard.
  3. Remove old copy: After traffic is confirmed to be routed to the new shard and the new shard is healthy, remove the old copy on shard 2.
This process keeps the application available throughout; only a tiny window of writes for a single bucket is paused during cutover. Because we move whole buckets rather than individual users, the total data moved during resharding is proportional to the number of buckets reassigned.
A dark schematic showing five database shards labeled Shard 0, Shard 1, Shard 2, Shard 3 and New Shard with arrows from user icons indicating data being copied and switched to the new shard. The top shows a four-step process: 1 copy, 2 catch up, 3 cutover, 4 remove.
Fixed buckets make resharding predictable and low-cost: users remain in the same bucket indefinitely, and resharding only updates the bucket-to-shard map. Pick a large bucket count (for example, thousands of buckets) so you can reassign small groups of users (fine-grained buckets), balance load precisely, and keep per-bucket traffic low during cutover.
Best practices and considerations
  • Choose the bucket count large enough to allow fine-grained rebalancing, but not so large that the bucket map becomes unwieldy. Thousands of buckets is a common starting point.
  • Ensure the copy and catch-up pipeline is robust: use change-data-capture (CDC), write-ahead-log replication, or a similar mechanism to stream incremental updates reliably.
  • Keep the cutover window as small as possible; plan for retries and client backoff during the short write pause.
  • Monitor bucket-level traffic and hot buckets. Very hot buckets can still require special handling (sharding within a bucket or application-level throttling).
Further reading and references

Watch Video