Rate Limiter Make It Scale

In production, API gateways are usually deployed as multiple instances behind a load balancer so requests don’t depend on a single gateway. A single gateway is a single point of failure and does not scale. However, naive local in-memory rate limits break in this distributed setup. Consider two gateways (A and B) and a user (Alan). The load balancer routes some of Alan’s requests to A and some to B. Each gateway only sees the requests routed to it and keeps a separate in-memory token bucket for Alan. That effectively gives Alan multiple buckets (one per gateway) and lets him exceed the intended global limit. Add a third gateway and he can make even more calls. In short: local in-memory counters lie in distributed systems. The reliable solution is to move buckets out of gateway memory into a single shared, fast store (for example, Redis) so all gateways read and update the same bucket for a given user. This centralization solves the multiple-buckets problem — but introduces another challenge: concurrent access and race conditions.

Why concurrency breaks naive shared counters

Accessing a token bucket involves three logical steps:

Read remaining tokens and the last-update timestamp.
Compute tokens to add based on elapsed time.
Write the new token count and timestamp.

If two gateways run these steps concurrently against the same Redis key, they can both read the same remaining tokens and both allow requests that should have been blocked. The read–compute–write sequence must be atomic to avoid this race.

Atomicity with Redis Lua scripts

Redis supports running Lua scripts atomically: while a script executes, no other client can mutate the keys it touches. Encapsulate the read–compute–write sequence in a single Lua script and run it with EVAL/EVALSHA. This guarantees correctness no matter how many gateways you have. Below is a compact, practical Lua script that implements a token-bucket check and update atomically. It assumes each bucket is a Redis hash with fields tokens and ts (timestamp in milliseconds).

-- token_bucket.lua
-- KEYS[1] = bucket key (e.g., "bucket:user:alan")
-- ARGV[1] = capacity (e.g., 50)
-- ARGV[2] = refill_rate_per_millisecond (e.g., 1.67/1000)
-- ARGV[3] = now (milliseconds)
-- ARGV[4] = tokens_requested (usually 1)

local key = KEYS[1]
local capacity = tonumber(ARGV[1])
local rate = tonumber(ARGV[2])
local now = tonumber(ARGV[3])
local req = tonumber(ARGV[4])

local data = redis.call("HMGET", key, "tokens", "ts")
local tokens = tonumber(data[1]) or capacity
local ts = tonumber(data[2]) or now

-- Refill tokens based on elapsed time
local elapsed = math.max(0, now - ts)
local added = elapsed * rate
tokens = math.min(capacity, tokens + added)

local allowed = 0
local retry_after = 0

if tokens >= req then
  tokens = tokens - req
  allowed = 1
else
  -- estimate milliseconds until next token available
  local missing = req - tokens
  retry_after = math.ceil(missing / rate)
end

-- Persist updated state
redis.call("HSET", key, "tokens", tokens, "ts", now)

return { allowed, tokens, retry_after }

A gateway should call this script for every API request. The script returns a three-element array:

allowed — 1 if allowed, 0 if rate-limited.
remaining tokens — the new token count.
retry_after — milliseconds until the next token (0 when allowed is 1).

Return position	Meaning	Example
1	`allowed` (0 or 1)	`1`
2	`remaining tokens`	`12.34`
3	`retry_after` (ms)	`500`

Note: Use integer millisecond timestamps (now) to avoid floating-point drift in production.

Decide and document behavior when the shared store is unavailable: do you fail open (allow traffic) or fail closed (reject traffic)? This is a critical trade-off—prioritize availability for general APIs and stricter protection for security-sensitive endpoints (for example, login or payment flows).

Operational considerations and trade-offs

Concern	Impact	Mitigation / Notes
Latency	Each request requires one Redis round-trip.	Co-locate Redis with gateways (same AZ/region) to keep RTT ≈ 1 ms. Avoid cross-region Redis on the critical path.
Throughput	Central Redis must handle script invocations per request.	A single Redis instance can handle ≈100k ops/s (workload dependent). If needed, use Redis Cluster or sharding.
High availability	Redis outage affects rate limiting.	Choose fail-open vs fail-closed, use Redis replication, persistence (AOF/RDB), and careful failover planning.
Atomicity	Prevents race conditions and double-allowing.	Run read–compute–write in a single Redis Lua script (atomic).
Cross-region redundancy	Adds latency if synchronous; eventual consistency if asynchronous.	Consider multi-region architecture carefully; avoid making synchronous cross-region calls in the request path.

Quick parameter examples

Parameter	Example
Capacity (burst)	`50` tokens
Refill rate	`1.67` tokens/second (≈`1.67/1000` per ms)
Typical use	capacity = 50; refill = 1.67/s => 100 req/min sustained

For security-critical APIs (e.g., login, password reset, anti-abuse paths), prefer failing-closed on store outages. For general product APIs where availability is more important, failing-open might be acceptable. Document and test your chosen mode.

Deployment and scaling recommendations

Run the Lua script via EVALSHA (load script once, call by SHA) to reduce overhead.
Keep Redis instances in the same availability zone or region as your gateways to minimize latency.
Monitor Redis operation latency, script execution time, and ops/s. Alert on degraded performance.
If you need higher throughput, shard keys by user ID across a Redis Cluster or partition users by consistent hashing.
Use replication and automated failover (Sentinel or managed Redis offerings). If you need cross-region failover, evaluate read/write latency and consistency implications.

Interview-friendly summary (talking points)

Put the rate limiter at the API gateway so you reject work early.
Count per account for authenticated users and per IP for anonymous traffic.
Use the token-bucket algorithm: capacity = burst, refill = sustained rate. Example: capacity 50; refill 1.67 tokens/s.
Store buckets centrally in Redis so all gateways operate on the same counters.
Make the check and update atomic by running the logic inside Redis as a Lua script. This prevents races when multiple gateways access the same bucket.
Return 429 Too Many Requests with a Retry-After header when rate-limited.
Decide and document fail-open vs fail-closed behavior for Redis outages based on endpoint sensitivity.
Key takeaway: local counters lie in distributed systems — authoritative counters must live in a shared store and be updated atomically.

When you need to guarantee a single resource is not double-assigned (concert seat, driver dispatch, etc.), apply the same pattern: keep authoritative state in one shared place and perform atomic updates. References and further reading:

Redis scripting: https://redis.io/docs/manual/programmability/eval-intro/
Token bucket algorithm: https://en.wikipedia.org/wiki/Token_bucket
Rate limiting design patterns: https://blog.cloudflare.com/rate-limiting-best-practices/

A dark-themed schematic of a ride-booking system showing a mobile app on the left, a load balancer routing requests to two services (labeled Leo and Maya), which connect to an app server and a database. The top shows numbered tabs for stages like "Clarify," "Estimate," and "High-level design."

The Interview, Replayed

Warm-Ups: The Read-Heavy Classics

Fan-Out: One Event, Many Readers

Rate Limiter Make It Scale

Why concurrency breaks naive shared counters

Atomicity with Redis Lua scripts

Operational considerations and trade-offs

Quick parameter examples

Deployment and scaling recommendations

Interview-friendly summary (talking points)

Watch Video

​Why concurrency breaks naive shared counters

​Atomicity with Redis Lua scripts

​Operational considerations and trade-offs

​Quick parameter examples

​Deployment and scaling recommendations

​Interview-friendly summary (talking points)

Watch Video

Why concurrency breaks naive shared counters

Atomicity with Redis Lua scripts

Operational considerations and trade-offs

Quick parameter examples

Deployment and scaling recommendations

Interview-friendly summary (talking points)