Skip to main content
In this lesson we design a rate limiter — a component that protects APIs and services from noisy clients who make too many requests in a short time. Rate limiting is essential for systems like public APIs, URL shorteners, and large web services (e.g., ChatGPT, Claude) to maintain performance and protect backend resources. We’ll use a simple six-step design framework. Step 1 is clarifying requirements; step 2 is a back-of-the-envelope estimation. The content below follows that sequence, with code samples and diagrams retained in their original position. Example attacker script (simple abusive client that posts repeatedly):
This could create millions of junk records and degrade service for normal users. We need a rate limiter that can slow down or block such noisy clients.

​
Step 1 — Clarify requirements (ask these questions)

  1. How do we identify who is making an API call?
Common identifiers:
  • Account ID (logged-in users)
  • IP address
  • API key
Each approach has trade-offs. Use the table below to summarize these trade-offs for quick interview answers. For this design exercise we assume:
  • Count per account for logged-in users.
  • Count per IP for non-logged-in requests.
  1. What is the limit per user?
For this example we’ll set a limit of 100 requests per minute per user.
In interviews, avoid pretending limits are production-ready. Say you would measure real traffic (requests/minute per user) and set thresholds to sit comfortably above typical usage while protecting the system.
  1. What should happen when the limit is exceeded?
Two main approaches:
  • Queue excess requests and process later.
  • Reject excess requests immediately.
Queuing can be useful in some contexts but introduces risk: clients often retry on slow or timed-out requests, which can create duplicates and extra load when the queue drains. For many APIs, rejecting excess requests with HTTP 429 and a Retry-After header is safer and clearer.
Queuing excess requests can lead to duplicate operations if clients retry while waiting. For most APIs, it’s preferable to reject excess requests with a clear HTTP 429 response and an explicit retry window.
Summary of scope for this design:
  • Limit: 100 requests per minute per user.
  • Identification: by account for logged-in users, otherwise by IP.
  • Behavior: reject excess requests with a clear response (HTTP 429 + Retry-After).
A dark-themed architecture diagram showing a user (labelled "alan" with IP 203.0.113.7) sending requests through a rate limiter (100/minute) to an app server and database. Notes on the right state limits are applied by account or IP, 100 per minute per user, and rejected requests include a reason.
Normal users should be unaffected by the rate limiter — it operates transparently for typical usage and only intervenes when a client exceeds the agreed-upon threshold.

​
Step 2 — Back-of-the-envelope estimation

Only estimate numbers that affect architecture choices (memory, throughput, peak QPS, etc.). Two key numbers for this design are: storage per user, and total request rate hitting the rate limiter.

​
1) Storage per user

A minimal per-user state for a fixed-window or sliding-window counter typically stores:
  • a numeric counter
  • a timestamp for the window start or last update
Estimated size: ~50 bytes per user (counter + timestamp + small metadata). Use this to reason about memory requirements. Example estimate:
  • 10,000,000 active users × 50 bytes/user ≈ 500 MB — small enough to store in memory on one reasonably sized server if desired, or sharded across multiple instances.
A dark-themed dashboard slide titled "Stored per user" showing a user's counter (Count: 42, Since: 10:41) and a storage estimate of 50 B/user scaled across 10,000,000 users with a 500 MB usage bar. Small "interviewer" and "you" user icons appear at the bottom.
Keep the ~50 B/user estimate in mind for capacity planning. If you use richer data (sliding logs, per-minute buckets), memory increases accordingly.

​
2) Request traffic (QPS estimate)

Example scenario:
  • 10,000,000 users, average 100 requests/day per user ⇒ 1,000,000,000 requests/day total.
Convert to QPS and peak:
  • 1,000,000,000 requests/day ≈ 11,574 requests/second (≈ 12k RPS) average.
  • If peak = 3× average, peak ≈ 35,000 RPS.
Use the table below for quick conversions you can present in an interview: Important clarification: the “100 requests/day per user” is an average activity metric — not the per-minute throttle. The per-minute limit (100/min) protects against short bursty abuse; normal users typically generate far fewer than 100 requests in a single minute.

​
Design implications (short checklist)

  • Memory: ~50 B/user (adjust if design uses buckets or logs).
  • Throughput: design rate limiter to handle average and peak RPS (e.g., 12k average, 35k peak in the example).
  • Placement: rate limiter should be on the request path (edge or API gateway) to protect downstream services.
  • Failure handling: fail-open vs fail-closed — prefer fail-open only if downstream can rate-limit again; otherwise fail-safe decisions depend on SLA.
  • Responses: use HTTP 429 + Retry-After to guide clients.
  • HTTP 429 Too Many Requests: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/429
  • Common rate limiting algorithms: Token Bucket, Leaky Bucket, Fixed Window, Sliding Window
  • Implementations and patterns: Redis-based counters, in-memory sharded counters, API gateways with built-in rate limiting
Use these talking points and numbers in interviews: they show you understand identifications, limits, client behavior trade-offs, storage and traffic estimation, and how to translate those estimates into capacity and placement decisions.

Watch Video