Rate limiting: what token bucket and leaky bucket really differ on
A token bucket permits bursts; a leaky bucket flattens traffic to a constant rate. Picking wrong shows up as throttled normal users, or a limit that does nothing.
The two classic rate limiting models get treated as synonyms, but their output shape is completely different: one preserves bursts, the other erases them.
Token bucket: bursts allowed
Tokens accumulate at a fixed rate into a bucket of capacity C. A request takes one token, and takes nothing if the bucket is empty.
- Rate R sets the long-run average throughput
- Capacity C sets how large a burst survives
After an idle period the bucket is full, so C requests pass instantly. That is required for normal behavior such as a page firing 20 requests at once. A leaky bucket spreads them out and the user watches things load one at a time.
Leaky bucket: constant output
Requests enter a queue and leave at a fixed rate. A full queue drops.
Output is constant, and the cost is added latency: requests wait in the queue. Stronger protection for the downstream, worse experience for the caller.
Comparison
| Dimension | Token bucket | Leaky bucket |
|---|---|---|
| Output rate | variable, up to C | constant |
| Bursts | allowed | not allowed |
| Adds latency | no | yes |
| Protects | services with headroom | fragile downstreams |
Choosing
Ask one question: can the downstream absorb a burst?
| Scenario | Pick | Why |
|---|---|---|
| Public API gateway | token bucket | real usage is bursty |
| Paid third-party API | leaky bucket | the vendor bills per second |
| Database connection pool | leaky bucket | a burst exhausts connections |
| Per-user frequency cap | token bucket | short dense activity is fine |
The distributed trap
Counting in process memory across several instances multiplies your limit by the instance count. Either share state (Redis INCR with expiry) or allocate limit / instanceCount to each.
Watch atomicity with Redis: a GET then SET races. Use INCR with EXPIRE, or one Lua script that does both.
Which status code
Return 429 Too Many Requests with a Retry-After telling the client how long to wait. A 429 without Retry-After makes clients retry blindly, which amplifies your own traffic.
A token bucket manages long-run average plus a burst budget; a leaky bucket manages constant output. Decide what your downstream can absorb first.
Comments
…