> ## Documentation Index
> Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Course Introduction
> Practical beginner course teaching data partitioning techniques like sharding, hashing, consistent hashing, and bucketing with hands-on labs and Python examples for stable, scalable distribution and rebalancing
This course is built for absolute beginners who want a practical, problem-driven introduction to data partitioning and distribution. Instead of teaching concepts in isolation, we introduce ideas like sharding, hash functions, consistent hashing, and bucketing precisely when they are needed to solve a real-world task. That way you learn the "why" and the "how" together, through short explanations and hands-on labs.
What you'll learn:
* Core concepts: sharding, hashing, consistent hashing, bucketing, and rebalancing.
* Practical skills: implement stable sharding, measure imbalance, and migrate shards with minimal disruption.
* Tools and techniques: deterministic hashing with Python, design patterns for partitioning, and strategies to reduce hot spots.
Hands-on labs are incremental so you can build understanding step by step while experimenting with code examples and small projects.
This course assumes no prior knowledge. Labs and examples are incremental, so you’ll build understanding step by step.
Hands-on example: basic sharding
* A simple way to partition identifiers across N shards is to hash the identifier and take the remainder modulo N.
* This approach is easy to implement and efficient, but it has limitations when the number of shards changes.
Example (naive modulo sharding):
```python theme={null}
# Assign an id to one of 4 shards
shard = hash(id) % 4
```
Important note about Python's built-in hash()
* Python’s built-in `hash()` is randomized per process for security reasons. That means the value of `hash(id)` can differ between runs, processes, or machines.
* For stable sharding (persisted assignments or cross-process consistency), use a deterministic hash function such as MD5, SHA1, or a non-cryptographic hash library like `xxhash`.
Deterministic shard example using hashlib.md5:
```python theme={null}
import hashlib
def stable_shard(id_str: str, num_shards: int) -> int:
digest = hashlib.md5(id_str.encode('utf-8')).hexdigest()
as_int = int(digest, 16)
return as_int % num_shards
# Example
shard = stable_shard("user:12345", 4)
```
Limitations of naive modulo sharding
* Rebalancing: changing `num_shards` (e.g., adding or removing shards) typically forces nearly all keys to move to different shards.
* Hotspots: uneven distribution of keys or traffic can overload certain shards.
* Operational complexity: migrating data and updating routing can be costly and time-consuming.
Advanced techniques introduced in the course
* Consistent hashing: reduces key movement when shards change by assigning keys and nodes to the same hash ring.
* Bucketing (or virtual nodes): increases distribution granularity by mapping each physical shard to multiple positions on the hash ring.
* Hybrid strategies: combine bucketing and consistent hashing or use rendezvous hashing for more predictable distribution.
Comparison: common partitioning strategies
| Strategy | When to use | Pros | Cons |
| - | - | - | - |
| `hash(id) % N` | Small, static clusters | Simple, fast | Most keys move on topology change |
| Consistent hashing | Dynamic clusters, frequent scaling | Minimizes key movement | More complex routing and implementation |
| Bucketing / virtual nodes | Large cardinality differences across shards | Smooths distribution, reduces hotspots | Extra indirection and bookkeeping |
Links and references
* [Hash functions — hashlib documentation (Python)](https://docs.python.org/3/library/hashlib.html)
* [Consistent hashing — Wikipedia](https://en.wikipedia.org/wiki/Consistent_hashing)
* [Rendezvous hashing (HRW) — Wikipedia](https://en.wikipedia.org/wiki/Rendezvous_hashing)
This course assumes no prior knowledge. Labs and examples are incremental, so you’ll build understanding step by step.
This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.