Rate Limiting

Why this matters

Every public API is one enthusiastic script away from disaster. A buggy client retrying in a tight loop, a scraper with no manners, a genuine flash crowd — without rate limiting, one actor's traffic becomes everyone's outage. Rate limiting is how you say "you get a fair share" in a way machines understand. It's also a business tool: free tier vs. paid tier is very often just two different rate limits wearing a pricing page.

Picture a nightclub bouncer with a velvet rope. The club holds 300 people comfortably. The bouncer lets people in at a steady pace — one out, one in — and turns away (politely, one hopes) anyone arriving faster than the club can absorb. Without the bouncer, the fire marshal shuts you down. Without rate limiting, your database is the fire marshal, and it expresses itself through timeouts.

Video: Rate Limiter System Design: Token Bucket, Leaky Bucket, Scaling — ByteByteGo
The classic toll-booth walkthrough — token bucket vs leaky bucket, and how they scale — the mental model every rate limiter builds on.

The algorithms

Token bucket

Each user has a bucket holding up to capacity tokens; tokens refill at rate per second. Each request costs one token. Bucket empty? Request rejected (HTTP 429 Too Many Requests).

flowchart LR
    T[Tokens refill<br/>10/sec] --> B[Bucket<br/>capacity 100]
    R[Request] --> B
    B -->|token available| OK[Allowed]
    B -->|empty| NO[429 Rejected]

Token bucket is the crowd favorite because it allows bursts: a user who's been quiet accumulates tokens and can then fire off 100 requests at once. Real-world traffic is bursty, so this matches reality. Large API providers — Stripe, GitHub, and AWS among them — publish rate-limit headers consistent with token-bucket behavior: quotas that refill over time and tolerate bursts.

Leaky bucket

Requests pour into a queue (the bucket) and drain out at a fixed rate. Bursts get smoothed rather than allowed: if the bucket overflows, requests are dropped. Where token bucket says "bursts are fine, you saved up for them," leaky bucket says "everyone marches at the same pace." Pick leaky bucket when downstream systems genuinely can't handle bursts — a fixed-rate pump, not a savings account.

Fixed vs. sliding window

The naive approach: "100 requests per minute," counted on minute boundaries. The flaw: at 11:59:59 a user fires 100 requests, at 12:00:00 the counter resets and they fire 100 more — 200 requests in two seconds, technically "within the limit." The sliding window fixes this by counting the last 60 seconds continuously. Sliding-window counters (often implemented in Redis as a sorted set of per-request timestamps) are the pragmatic default for per-user API limits.

flowchart TD
    F["Fixed window<br/>100 req at 11:59:59 + 100 at 12:00:00<br/>= 200 in 2s — allowed! ⚠️"]
    S["Sliding window<br/>counts trailing 60s<br/>= 200 in 60s — rejected ✓"]

Video: Rate Limiting Explained: Token Bucket vs Sliding Window | System Design By Microsoft SWE — Mayank Joshi
Step-by-step on token bucket and leaky bucket — the two classic limiting algorithms.

Distributed rate limiting

One server can count in memory. Ten servers behind a load balancer cannot — each sees only a tenth of a user's traffic, so each allows ten times too much. The standard fix: a shared sliding window in Redis, built from a sorted set of per-request timestamps:

now = current timestamp in ms
window_start = now - 60_000

ZREMRANGEBYSCORE ratelimit:{user_id} -inf window_start   # evict anything older than 60s
count = ZCOUNT ratelimit:{user_id} window_start now      # requests in the trailing 60s
if count >= 100: reject with 429                        # rejected requests are NOT logged
ZADD ratelimit:{user_id} now {now}:{uuid}                # allowed: log this request
EXPIRE ratelimit:{user_id} 60                            # don't let idle keys pile up

Order matters: check before you log. If you logged the request first and then rejected it, every rejected request would still occupy a slot in the window — a burst of junk traffic would keep the user's window full long after the burst ended, punishing them for the attack. Two more details worth knowing: the member is now plus a unique suffix, because two requests landing in the same millisecond would otherwise collapse into one sorted-set entry and undercount; and in production this check-and-add runs as a single Lua script so two servers can't both read count = 99 and both allow.

No minute boundaries, no reset moments — every request is judged against the trailing 60 seconds, which is what kills the 11:59:59 → 12:00:00 trick from two sections ago. One honest caveat: the simpler INCR + EXPIRE version of this snippet is a fixed window, and it carries exactly that boundary exploit. If you copy the fixed-window version anywhere, you now know precisely what you're buying.

This works beautifully until Redis becomes the thing you're protecting from — at very high scale, teams use local buckets with periodic sync, or probabilistic structures, accepting slightly fuzzy limits to avoid a central counter on the hot path. Precision is a spectrum; "roughly 100 requests per minute" is usually fine.

Video: Designing a Distributed Rate Limiter (Under 20 Mins) | System Design Interview — ByteScaler
Designs a distributed rate limiter end to end — the interview-ready walkthrough.

Being a good citizen about it

Rate limiting isn't just enforcement — it's communication. Well-behaved APIs return:

• 429 status when the limit bites (not 403, not 500 — 429 means "slow down and retry").
• Retry-After header saying how long to wait.
• X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset headers so clients can pace themselves before hitting the wall.

And on the client side: respect 429s with exponential backoff and jitter (retry after 1s, 2s, 4s… plus randomness so a thousand clients don't retry in lockstep — the thundering herd's needy cousin). A client that ignores backoff signals is how you get rate-limited harder.

Video: System Design Mock Interview: Design a Rate Limiter (with Meta Engineering Manager) — Exponent
Watch a real Meta engineering manager design a rate limiter out loud — the closest you'll get to seeing the toll booth engineered from the inside.

Where to enforce

Defense in depth, from the edge inward:

1. CDN / edge (Cloudflare, AWS WAF) — kill obvious abuse (DDoS, scrapers) before it costs you compute.
2. API gateway — per-key quotas and tiers; this is the natural home (see API Gateways).
3. Service level — protect expensive endpoints individually; your /search endpoint deserves a stricter limit than /health.

One more subtlety: rate-limit by something the abuser can't trivially rotate. IP-based limits fall to botnets with a million IPs; authenticated-user or API-key limits hold up better. And always leave headroom for your own health checks — nothing is sadder than your monitoring getting 429'd during the outage it's trying to report.

Video: 30ms of Clock Drift Doubled the Rate Limit — Design a Rate Limiter — TheCodeForge
Shows rate-limiter placement trade-offs — edge versus app — with a real clock-drift bug.

Takeaways

1. Token bucket allows saved-up bursts; leaky bucket enforces a steady pace; sliding windows fix the boundary exploit.
2. Distributed systems need shared counters (Redis) — local counters let each server allow the full limit.
3. Communicate limits with 429 + Retry-After + X-RateLimit-* headers; clients should back off exponentially with jitter.
4. Enforce in layers (edge → gateway → service) and limit by API key or user, not just IP.

Go deeper

Want to keep pulling this thread? These talks and tutorials go further than we did here:

• Rate Limiter System Design: Token Bucket, Leaky Bucket, Scaling — Alex Xu, ByteByteGo. Token bucket, leaky bucket, sliding window, and scaling across servers.
• Token bucket vs leaky bucket, rate limiting explained — Konadu (~11 min). The two classic algorithms, the fixed-window exploit, and sliding windows.
• Rate Limiter Explained: 4 Algorithms + Interview Q&A — The Leap (~20 min). From fixed window to token bucket, then the multi-server problem: Redis races and 429s.
• Rate Limiter: Low Level System Design — Gaurav Sen. A dedicated rate-limiter design walkthrough from the most-cited system-design educator on YouTube — the long-form companion to this lesson.
• Design a Distributed Rate Limiter w/ a Ex-Meta Staff Engineer: System Design Breakdown — Hello Interview. An ex-Meta staff engineer designs a distributed rate limiter end to end — algorithms, Redis races, and multi-server scaling.

Sources & further reading

• Stripe API documentation, "Rate limits" — rate limits and headers in a production API.
• GitHub REST API documentation, "Rate limits" — primary/secondary limits and how to respect them.
• Cloudflare Learning Center, "What is rate limiting?" — edge enforcement and DDoS context.
• AWS Architecture Blog, "Rate limiting with API Gateway" / token-bucket via Redis patterns — distributed counting implementations.