Course Introduction

This course is built for absolute beginners who want a practical, problem-driven introduction to data partitioning and distribution. Instead of teaching concepts in isolation, we introduce ideas like sharding, hash functions, consistent hashing, and bucketing precisely when they are needed to solve a real-world task. That way you learn the “why” and the “how” together, through short explanations and hands-on labs. What you’ll learn:

Core concepts: sharding, hashing, consistent hashing, bucketing, and rebalancing.
Practical skills: implement stable sharding, measure imbalance, and migrate shards with minimal disruption.
Tools and techniques: deterministic hashing with Python, design patterns for partitioning, and strategies to reduce hot spots.

Hands-on labs are incremental so you can build understanding step by step while experimenting with code examples and small projects. This course assumes no prior knowledge. Labs and examples are incremental, so you’ll build understanding step by step. Hands-on example: basic sharding

A simple way to partition identifiers across N shards is to hash the identifier and take the remainder modulo N.
This approach is easy to implement and efficient, but it has limitations when the number of shards changes.

Example (naive modulo sharding):

# Assign an id to one of 4 shards
shard = hash(id) % 4

Important note about Python’s built-in hash()

Python’s built-in hash() is randomized per process for security reasons. That means the value of hash(id) can differ between runs, processes, or machines.
For stable sharding (persisted assignments or cross-process consistency), use a deterministic hash function such as MD5, SHA1, or a non-cryptographic hash library like xxhash.

Deterministic shard example using hashlib.md5:

import hashlib

def stable_shard(id_str: str, num_shards: int) -> int:
    digest = hashlib.md5(id_str.encode('utf-8')).hexdigest()
    as_int = int(digest, 16)
    return as_int % num_shards

# Example
shard = stable_shard("user:12345", 4)

Limitations of naive modulo sharding

Rebalancing: changing num_shards (e.g., adding or removing shards) typically forces nearly all keys to move to different shards.
Hotspots: uneven distribution of keys or traffic can overload certain shards.
Operational complexity: migrating data and updating routing can be costly and time-consuming.

Advanced techniques introduced in the course

Consistent hashing: reduces key movement when shards change by assigning keys and nodes to the same hash ring.
Bucketing (or virtual nodes): increases distribution granularity by mapping each physical shard to multiple positions on the hash ring.
Hybrid strategies: combine bucketing and consistent hashing or use rendezvous hashing for more predictable distribution.

Comparison: common partitioning strategies

Strategy	When to use	Pros	Cons
`hash(id) % N`	Small, static clusters	Simple, fast	Most keys move on topology change
Consistent hashing	Dynamic clusters, frequent scaling	Minimizes key movement	More complex routing and implementation
Bucketing / virtual nodes	Large cardinality differences across shards	Smooths distribution, reduces hotspots	Extra indirection and bookkeeping

Links and references

This course assumes no prior knowledge. Labs and examples are incremental, so you’ll build understanding step by step.

Why One Database Isn't Enough

Sharding Fundamentals

Routing and Hot Shards

Shard Keys and Partitioning

Resharding and Cross-Shard Queries

Course Introduction

Watch Video