Low-Level Design: A Rate Limiter Library

Designing a rate limiter from requirements to API: token bucket against fixed window with the boundary burst measured, an injected clock, thread safety, and what .NET, Java, Go and Rust’s libraries really do.

A rate limiter is the perfect low-level design exercise. It’s small enough to write in an afternoon, it has a real algorithm choice, it needs a clock, it needs to be thread-safe, and the naive version has a bug you can measure. It’s also asked in interviews constantly, usually badly: “design a rate limiter” with no mention of what happens at the window edge, who the limit applies to, or how anyone tests it.

We’ll design one properly: requirements first, then the algorithm with the failure demonstrated in running code, then the interface, the clock, the concurrency, and finally what the four ecosystems’ own libraries chose, which is more varied than the blog posts suggest.

Try this first

You limit each customer to 5 requests a second by counting requests in each calendar second and resetting the counter when the second ticks over.

A client sends 5 requests at 12:00:00.900 and 5 more at 12:00:01.000. How many get through, and over what period of time? Write it down.

Requirements before algorithms

The questions worth asking before any code:

1. What are we protecting? A downstream database, a third-party API with its own quota, CPU on this process, or fairness between tenants. The answer decides where the limiter lives and what the limit counts.
2. What’s the unit? Requests, bytes, or “cost” points. Real APIs charge more for expensive calls, which means the limiter’s Allow needs a count, not a boolean question.
3. Who is limited? Per API key, per user, per IP, per tenant, or globally. This is the partition key, and it’s a first-class design axis: .NET’s PartitionedRateLimiter<TResource> takes a function that derives the key from the request.
4. What happens when the limit is hit? Reject immediately, queue and wait, or degrade. A library that only rejects can’t serve a client that would rather wait 50 ms.
5. What does the caller learn? “No” is not enough. A client needs to know when to try again.
6. One process or many? In-process limiting is arithmetic. Distributed limiting is a consensus problem with a budget (Part 30).

One more thing worth knowing before you choose: the cheapest limiter usually does nearly all the work. Stripe, writing in 2017 about the four limiters in front of their API, reported that the plain request-rate limiter rejected millions of requests a month, the concurrency limiter about 12,000, and the worker-utilization load shedder around 100.

The algorithm, and the bug in the obvious one

Fixed window is the one everyone writes first: count requests in the current second, reset on the boundary. It’s simple, it’s cheap, and it allows twice the limit across a boundary.

Token bucket holds a bucket of tokens that refills at a steady rate up to a capacity. Each request takes one. The capacity is the burst you tolerate; the refill rate is the sustained rate you allow.

Our lab implements both, in all four languages, behind an injected clock, and replays the same request times. Because the clock is injected, all four languages printed the same decisions. The boundary scenario is the one from “Try this first”: five requests at 900 ms, five at 1000 ms.

Request at	Token bucket (5 capacity, 5/s)	Fixed window (5 per second)
900 ms	allowed	allowed
900 ms	allowed	allowed
900 ms	allowed	allowed
900 ms	allowed	allowed
900 ms	allowed	allowed
1000 ms	denied	allowed
1000 ms	denied	allowed
1000 ms	denied	allowed
1000 ms	denied	allowed
1000 ms	denied	allowed

Totals: the fixed window allowed 10 of 10 requests in 100 ms; the token bucket allowed 5.

Ten requests in 100 ms, against a limit of five a second. The token bucket, with the same average rate, allowed five and refused the rest, because by 1000 ms only 0.5 tokens had refilled.

That’s the whole argument for token bucket in one table. Fixed window is easy to implement and easy to abuse; token bucket costs a subtraction and a multiplication and behaves under attack.

Two more options worth naming:

• Sliding window log keeps every request’s timestamp and counts the ones inside the window. Exact, and expensive: Cloudflare rejected it because storing timestamps “has huge processing and memory requirements”.
• Sliding window counter weights the previous window’s count by how far into the current one you are. Cloudflare measured theirs against 400 million requests from 270,000 sources: 0.003% of requests were wrongly allowed or limited, with a 6% average difference between the approximate rate and the real one. They chose it, in 2017, because of their datastore: “We were constrained to use the memcached protocol and this algorithm requires multiple distinct operations that we cannot do atomically”.

And a naming trap. “Token bucket versus leaky bucket” is usually a false dichotomy. ITU-T’s specification of the Generic Cell Rate Algorithm gives two algorithms, a virtual scheduling algorithm and a continuous-state leaky bucket, and states that they “determine the same cells to be conforming and thus the same cells to be non-conforming”. A leaky bucket used as a meter is a token bucket turned inside out. A leaky bucket used as a queue, which is what some articles mean, is a different thing entirely: it smooths output rather than metering input.

The core, in four languages

The interesting part is what the type signature says. Here’s the Go version from our lab:

// Clock is the seam: tests pass a fake, production passes time.Now.
type Clock func() time.Time

type TokenBucket struct {
	capacity   float64
	perSecond  float64
	tokens     float64
	lastRefill time.Time
	now        Clock
}

func NewTokenBucket(capacity int, perSecond float64, now Clock) *TokenBucket {
	return &TokenBucket{capacity: float64(capacity), perSecond: perSecond,
		tokens: float64(capacity), lastRefill: now(), now: now}
}

func (b *TokenBucket) Allow() bool {
	t := b.now()
	// max(0, ...): a clock that jumps backwards must not mint tokens.
	elapsed := max(0, t.Sub(b.lastRefill).Seconds())
	b.lastRefill = t
	b.tokens = min(b.capacity, b.tokens+elapsed*b.perSecond)
	if b.tokens >= 1 {
		b.tokens--
		return true
	}
	return false
}

Three design decisions are already visible.

Lazy refill. There’s no timer and no background goroutine. Tokens are computed from elapsed time when someone asks. A million idle buckets cost nothing. RFC 2698, specifying a two-rate meter, blesses this directly: “The actual implementation of a Meter doesn’t need to be modeled according to the above formal specification.”

The clock is a parameter. Not time.Now() inside the method. This is what makes the boundary test above a unit test rather than a sleep. Note the max(0, ...): a clock that jumps backwards must not mint tokens.

One thing these samples leave out on purpose: a lock. They’re written single-threaded so the arithmetic is visible; the concurrency section below says where the mutex goes.

Allow returns a bool, which is already a simplification. RFC 2697’s meter is three-valued: a packet is green if it doesn’t exceed the committed burst size, “yellow if it does exceed the CBS, but not the EBS”, and red otherwise. Most real APIs want that middle state, for soft limits and warnings. It also takes no cost: a real API charges more for expensive calls, so the method wants an Allow(int cost), which is what AcquireAsync(permitCount) and AllowN(n) give you.

C#: TimeProvider is the clock seam

var clock = new TestClock();
var bucket = new TokenBucket(capacity: 5, perSecond: 5, clock);
Console.WriteLine($"at 0 ms: {Allowed(6)} of 6 allowed");
clock.Advance(TimeSpan.FromMilliseconds(1000));
Console.WriteLine($"a second later: {Allowed(6)} of 6 allowed");

int Allowed(int requests)
{
    var allowed = 0;
    for (var i = 0; i < requests; i++)
    {
        if (bucket.Allow())
        {
            allowed++;
        }
    }
    return allowed;
}

// A TimeProvider the test moves by hand. TimeProvider arrived in .NET 8 for exactly this.
sealed class TestClock : TimeProvider
{
    private long _ticks;

    public void Advance(TimeSpan by) => _ticks += by.Ticks;

    public override DateTimeOffset GetUtcNow() => DateTimeOffset.UnixEpoch.AddTicks(_ticks);
}

sealed class TokenBucket(int capacity, double perSecond, TimeProvider clock)
{
    private double _tokens = capacity;
    private DateTimeOffset _lastRefill = clock.GetUtcNow();

    public bool Allow()
    {
        var now = clock.GetUtcNow();
        var elapsed = Math.Max(0, (now - _lastRefill).TotalSeconds);
        _lastRefill = now;
        _tokens = Math.Min(capacity, _tokens + elapsed * perSecond);
        if (_tokens < 1)
        {
            return false;
        }
        _tokens--;
        return true;
    }
}

It prints:

at 0 ms: 5 of 6 allowed
a second later: 5 of 6 allowed

TimeProvider is the .NET 8 abstraction for “the clock”, and Microsoft.Extensions.TimeProvider.Testing ships a FakeTimeProvider so you don’t have to write TestClock yourself.

Java: java.time.Clock, with a caveat

Java has had an injectable clock since Java 8, and its javadoc states the intent: “The primary purpose of this abstraction is to allow alternate clocks to be plugged in as and when required.” Our lab’s Java bucket takes one.

The caveat matters for a rate limiter, though: java.time.Clock is a wall clock. It can jump backwards when NTP corrects it or an operator changes the time, and Guava’s RateLimiter doesn’t accept one at all. If you write your own in Java, measure elapsed time with System.nanoTime() and inject a seam around that instead. Our lab’s Java version uses Clock because the fake is easy to read, and it is the wrong choice for production.

The clock itself is the interesting part:

    // A Clock the test moves by hand. java.time.Clock exists for exactly this.
    static final class TestClock extends Clock {
        private Instant now = Instant.EPOCH;

        void setMillis(long ms) {
            now = Instant.EPOCH.plusMillis(ms);
        }

        @Override
        public ZoneOffset getZone() {
            return ZoneOffset.UTC;
        }

        @Override
        public Clock withZone(java.time.ZoneId zone) {
            return this;
        }

        @Override
        public Instant instant() {
            return now;
        }
    }

Rust: a trait, and a monotonic clock with rules

Rust’s Instant is documented as monotonic (“An instant may jump forwards or experience time dilation […] but it will never go backwards”), and its subtraction methods “saturate to zero” rather than panicking on a backwards jump. Go’s x/time/rate hand-writes the same guard.

That’s the general rule for any limiter: use a monotonic clock, and clamp negative elapsed time to zero, or a clock correction hands every client a free burst.

use std::cell::Cell;
use std::time::Duration;

trait Clock {
    fn now(&self) -> Duration;
}

struct TestClock {
    elapsed: Cell<Duration>,
}

impl Clock for TestClock {
    fn now(&self) -> Duration {
        self.elapsed.get()
    }
}

struct TokenBucket<'a, C: Clock> {
    capacity: f64,
    per_second: f64,
    tokens: f64,
    last_refill: Duration,
    clock: &'a C,
}

impl<'a, C: Clock> TokenBucket<'a, C> {
    fn new(capacity: u32, per_second: f64, clock: &'a C) -> Self {
        TokenBucket {
            capacity: f64::from(capacity),
            per_second,
            tokens: f64::from(capacity),
            last_refill: clock.now(),
            clock,
        }
    }

    fn allow(&mut self) -> bool {
        let now = self.clock.now();
        // saturating_sub is the guard: a clock that goes backwards adds no tokens.
        let elapsed = now.saturating_sub(self.last_refill).as_secs_f64();
        self.last_refill = now;
        self.tokens = (self.tokens + elapsed * self.per_second).min(self.capacity);
        if self.tokens < 1.0 {
            return false;
        }
        self.tokens -= 1.0;
        true
    }
}

fn main() {
    let clock = TestClock {
        elapsed: Cell::new(Duration::ZERO),
    };
    let mut bucket = TokenBucket::new(5, 5.0, &clock);
    let allowed = |bucket: &mut TokenBucket<TestClock>| (0..6).filter(|_| bucket.allow()).count();
    println!("at 0 ms: {} of 6 allowed", allowed(&mut bucket));
    clock.elapsed.set(Duration::from_millis(1000));
    println!("a second later: {} of 6 allowed", allowed(&mut bucket));
}

It prints:

at 0 ms: 5 of 6 allowed
a second later: 5 of 6 allowed

The interface: what “no” should say

A boolean tells a client nothing. The better shapes, in order of usefulness:

1. Allow or deny, plus when to retry. Retry-After in HTTP terms, a TimeSpan in code.
2. Reserve. Go’s x/time/rate returns a Reservation that tells you how long to wait, and lets you cancel it and give the token back.
3. Wait. The caller says “I’ll wait up to 100 ms”, and the limiter queues them.

.NET models all of this with a lease object:

• AttemptAcquire(int permitCount = 1) is the “fast synchronous attempt”; AcquireAsync waits.
• Both return a RateLimitLease, and a rejection is still a lease: IsAcquired is false, and metadata such as RetryAfter hangs off the same object.
• The lease is disposable, and disposing it is how the limiter learns you’re done with the permits.
• permitCount: 0 is a documented probe: on AttemptAcquire, “Set permitCount to 0 to get whether permits are exhausted”.

Go’s package documentation puts the guidance in one line: of Allow, Reserve and Wait, “most callers should use Wait”. The reason is in the implementation: a cancelled Wait returns the token to the bucket, because it holds a reservation it can cancel. Allow has nothing to give back.

Concurrency: one lock, held briefly

A rate limiter is shared mutable state with a tiny critical section, which is exactly the shape from Part 11. Take a mutex, do the arithmetic, release.

What the real libraries do:

• Go’s x/time/rate uses a plain sync.Mutex on every call, and documents that a Limiter “is safe for simultaneous use by multiple goroutines”.
• Guava’s RateLimiter warns in its second paragraph: “RateLimiter is safe for concurrent use: It will restrict the total rate of calls from all threads. Note, however, that it does not guarantee fairness.”
• Bucket4j documents a lock-free default, and Rust’s governor keeps its state in an AtomicU64.

Unless you’ve measured a contention problem, the lock is right: the critical section is a few arithmetic operations, and per-key partitioning spreads the contention anyway. That’s Part 11’s lesson applied: correctness first, and shard before you go lock-free.

What the libraries actually chose

Four ecosystems, four different algorithms, one name:

Library	Algorithm	Clock seam	Notes
.NET System.Threading.RateLimiting	token bucket, fixed and sliding window, concurrency	AutoReplenishment = false plus TryReplenish()	leases, queueing, partitioning
Guava RateLimiter (Java)	smoothed permit spacing	none (internal stopwatch)	@Beta since 13.0, no fairness guarantee
golang.org/x/time/rate	token bucket	AllowN(t, n)-style method twins	Allow, Reserve, Wait
Rust governor	GCRA	a clock trait, with a fake for tests	lock-free

Four details from their documentation that contradict what people expect:

• .NET’s rate limiter never mentions TimeProvider. Microsoft shipped a clock abstraction in .NET 8 and a rate limiter beside it, and didn’t connect them. The test seam is turning off the timer and pumping it by hand, which our lab does:

attempt 1: allowed
attempt 2: allowed
attempt 3: allowed
attempt 4: allowed
attempt 5: allowed
attempt 6: denied
TryReplenish immediately: returned True, available 0
TryReplenish after 250 ms: returned True, available 1
one attempt after that: allowed
available now: 0

Note the two TryReplenish lines: the call returns true but adds nothing until ReplenishmentPeriod has genuinely elapsed on the real clock. The manual pump is a seam, not a fake clock.

• ASP.NET Core’s rate limiting middleware rejects with 503, not 429. RejectionStatusCode “Defaults to Status503ServiceUnavailable”, and the docs show you how to change it. Out of the box it doesn’t implement RFC 6585.
• Guava’s cost model is backwards from the obvious one: “an invocation to acquire(1) and an invocation to acquire(1000) will result in exactly the same throttling, if any […] it is the next request that will experience extra throttling, thus paying for the cost of the expensive task.”
• Each auto-replenishing .NET limiter owns a timer. The ASP.NET documentation states it plainly: constructing a TokenBucketRateLimiter with AutoReplenishment set to true “gives each limiter instance its own timer”, while its own helpers “replenish all of their limiters from a single shared timer”. Create one per request and you are creating timers per request.

The HTTP side: 429, and headers that aren’t standard yet

RFC 6585 defines 429 in a handful of sentences, and it’s worth reading how little it says: “The 429 status code indicates that the user has sent too many requests in a given amount of time (‘rate limiting’). The response representations SHOULD include details explaining the condition, and MAY include a Retry-After header indicating how long to wait before making a new request.” It explicitly declines to define identity or counting: “this specification does not define how the origin server identifies the user, nor how it counts requests.” The only hard requirement is that a 429 “MUST NOT be stored by a cache”.

The RateLimit-* headers everyone copies are not a standard. As of September 2026 they’re at draft-ietf-httpapi-ratelimit-headers-11, dated 23 May 2026, with an intended RFC status of “None” and an early directorate review of the previous revision marked “Not ready”. The design has changed, too: the three headers in every blog post (RateLimit-Limit, RateLimit-Remaining, RateLimit-Reset) have been replaced by two structured fields, RateLimit-Policy and RateLimit.

The draft is also clear that its numbers are hints rather than a promise, and that when both are present the “Retry-After field MUST take precedence”. So: return 429 with Retry-After, document your own headers, and don’t promise clients a standard that doesn’t exist. GitHub, for what it’s worth, returns 403 or 429 for its primary rate limit and ships x-ratelimit-* headers of its own.

Explain it like I’m ten

A bus company sells tickets from a machine that holds five at a time and adds one more every twelve seconds, so five a minute.

• If nobody has been for a while, the machine is full: five people can travel at once. That’s your burst.
• After that, one person can go every twelve seconds. That’s your rate.
• The machine never holds more than five, however long it sits idle. That’s the cap that stops a week of saved-up tickets arriving at once.

The broken version counts tickets per minute and forgets at the top of each minute: five people at 10:59:59 and five more at 11:00:00, and ten of them travel in the same two seconds.

The precise version

• The machine is the bucket, the tickets are tokens, five is the capacity (the burst) and one per twelve seconds is the refill rate.
• Forgetting at the top of the minute is the fixed window, and its failure at the boundary is what our lab measured.
• “Nobody came for a while” is why the cap matters: without it, idle time turns into an unbounded burst.
• Where the analogy breaks: the machine doesn’t need to add tokens on a timer. The code works out how many would have appeared since the last request, which is why an idle limiter costs nothing.

Trade-offs

• Token bucket allows bursts on purpose. If your downstream can’t take capacity requests at once, capacity is too big. Set it to what the protected thing survives, not to a round number.
• Fixed window is cheaper and lies at the edges. For a coarse abuse limit (“no more than 10,000 a day”) that’s fine; for protecting a database it isn’t.
• Rejecting is cheap, queueing is a design. A queue needs a bound, an order and a timeout, or it becomes an unbounded buffer that turns a rate problem into a memory problem (Part 11).
• Per-key limiting needs eviction. One bucket per API key is a map that grows forever unless idle entries are removed.
• In-process limiting is per-instance. Ten replicas with “100 a second” each allow 1,000. Either divide the budget, or move the counter to a shared store and accept the round trip (Part 30).
• Fail open or closed, decided in advance. Stripe’s rule for their Redis-backed limiters: “if Redis were to go down, requests wouldn’t be affected […] fail open and the API would still stay functional.”

Common mistakes

• Counting in calendar windows. The boundary burst above. If you must, use a sliding window counter.
• Calling time.Now() inside the limiter. Now the only way to test refill is sleep, and your test suite gets slower and flakier.
• Using a wall clock. An NTP correction or a leap second becomes a free burst, or a lockout. Use monotonic time and clamp negative elapsed time to zero. Monotonic isn’t quite enough either: Rust’s Instant documentation says it is monotonic but not steady, and “it is also not specified whether system suspends count as elapsed time or not”, so a laptop waking from sleep may owe you tokens it never grants.
• Returning only a boolean. Clients then retry immediately, which is the worst possible behaviour. Return when to retry.
• One global lock for every key. Partition by key so unrelated tenants don’t contend.
• Limiting at the wrong layer. A limiter behind an expensive authentication step still pays for the expensive step. Put the cheap limiter first, and mind the order in chained limiters: .NET’s docs warn that time-based limiters “don’t return a permit that was already acquired when a later limiter in the chain rejects the request”.
• Forgetting the retry storm. Every rejected client retrying at the same moment reproduces the spike. Tell them when, and add jitter.
• Assuming the RateLimit-* headers are standard. They’re a draft, and the field names changed.

Interview questions

Try to answer each one before opening the model answer.

1. Design a rate limiter for an API. Start with the questions.

Show a strong answer

• Ask first: what are we protecting, what’s the unit (requests, bytes, cost), who is limited (key, user, IP, tenant), what happens on rejection (deny, queue, degrade), and is this one process or many?
• Then choose: token bucket for burst plus sustained rate; fixed window only for coarse quotas; sliding window counter when a shared store constrains you.
• The interface: Allow(cost) returning a decision plus a retry time, with an optional wait, and a partition key.
• The clock: injected, monotonic.
• The limits themselves: configurable per key, with a default, and observable (metrics for allowed, rejected, and remaining).

Likely follow-up: “Where does it run?” In-process for protecting this instance, at the gateway for protecting everything behind it, and in a shared store when the quota is global.

2. Why not a fixed window?

Show a strong answer

• The boundary burst: a client can send the full limit at the end of one window and again at the start of the next. Our lab: five at 900 ms and five at 1000 ms, all ten allowed by a 5-per-second fixed window; the token bucket allowed five.
• Effectively double the rate over a short span, which is exactly when a downstream falls over.
• When it’s fine: coarse quotas over long windows, where a doubled burst is harmless.
• The alternatives: token bucket in-process; a sliding window counter if you need cheap shared-state counting.

Likely follow-up: “How does the sliding window counter work?” Weight the previous window’s count by the fraction of the current window elapsed. Cloudflare measured 0.003% of requests wrongly classified across 400 million requests.

3. How do you make it testable?

Show a strong answer

• Inject the clock. A TimeProvider in .NET, a Clock in Java, a func() time.Time or an …At(t) method in Go, a trait in Rust.
• Then the tests are arithmetic: advance the fake clock, assert the decisions, with no sleeps and no flakiness. Our lab replays the same times in four languages and gets identical output.
• Test the boundaries: empty bucket, full bucket, exactly one token, a long idle period (does it cap?), and a backwards clock jump.
• If the library doesn’t take a clock: .NET’s own limiter doesn’t; it gives you AutoReplenishment = false plus a manual TryReplenish() instead.

Likely follow-up: “How do you test the concurrent path?” Hammer it from N threads and assert the total allowed is within the expected bound, and run it under the race detector where you have one.

4. How would you limit per API key without leaking memory?

Show a strong answer

• A map from key to bucket, created on first use. .NET’s PartitionedRateLimiter<TResource> does this for you, with a function that derives the key from the request.
• Eviction: an LRU or an idle timeout, because a bucket that’s full and untouched is indistinguishable from no bucket at all.
• Sharding the lock: per-key locks or a concurrent map, so unrelated keys don’t contend.
• Watch the creation race: two threads asking for the same new key must end up with one bucket. Java’s computeIfAbsent runs the factory once; .NET’s GetOrAdd may run it more than once (Part 11).

Likely follow-up: “What if one key is enormously hot?” Shard that key across instances, or promote it to its own limiter with its own budget.

5. How does distributed rate limiting differ?

Show a strong answer

• The state moves out of process, so every decision is a network round trip, and the counter needs an atomic operation (INCR plus expiry, or a Lua script).
• Approximation is the norm: exact counting is too expensive, which is why Cloudflare chose a sliding window counter with a measured 6% average error against the true rate.
• Failure policy matters more than accuracy: decide fail-open or fail-closed before the store goes down. Stripe’s limiters fail open.
• Cheap local limits first: an in-process limiter absorbs the obvious abuse; the shared one enforces the real quota.
• Clock skew across nodes makes window edges fuzzy: prefer counters with TTLs over timestamps compared across machines.

Likely follow-up: “How do you divide a global limit across instances?” Static division is simple and wastes capacity; a shared store is accurate and costs a hop; a token-lease scheme, where instances check out chunks of the budget, is the middle ground.

6. What should the API return when it rejects?

Show a strong answer

• 429 Too Many Requests, with Retry-After. RFC 6585 makes the header a MAY, but a client with no retry hint retries immediately.
• A body that says which limit was hit and for which key, without leaking other tenants’ data.
• Headers for the remaining budget are useful, but know that RateLimit-* is still an Internet-Draft, and its field design changed to RateLimit-Policy and RateLimit.
• In code: return a value that carries the reason and the wait, not a bare boolean. .NET returns a lease whose IsAcquired is false and which carries RetryAfter metadata.
• Watch your framework’s default: ASP.NET Core’s middleware rejects with 503 unless you set 429.

Likely follow-up: “Should the retry time be exact?” Give a floor, and tell clients to add jitter, or every rejected client returns at the same instant.

7. Where should the limiter live?

Show a strong answer

• At the edge (CDN, gateway) for abuse and volumetric protection, before the request costs you anything.
• In the service for per-tenant fairness and for protecting a specific downstream.
• Around the client of a third-party API, to stay inside someone else’s quota, which is usually a wait-style limiter rather than a rejecting one.
• Layered is normal: a cheap global limit, then per-key, then per-expensive-operation. Order them cheapest first.
• Not in the database: counting in the thing you’re protecting doesn’t protect it.

Likely follow-up: “Where do you put the limit for a login endpoint?” At the edge by IP for volumetric abuse, and per account for credential stuffing, with a longer window and a lockout policy.

8. Your limiter is allowing more than the configured rate. How do you debug it?

Show a strong answer

• Count instances first: N replicas × the limit is the usual answer, not a bug in the algorithm.
• Check the window semantics: a fixed window allows a double burst at the boundary by design.
• Check the clock: a wall clock that jumps backwards produces free tokens; elapsed time must be clamped at zero.
• Check the key: if the partition key is wrong (per connection instead of per user, say), each client gets its own budget.
• Check the ordering: an authenticated limiter placed after an expensive step lets the expensive step run anyway, and chained limiters can consume permits for requests that are later rejected.
• Then measure: log allowed and rejected counts per key, and replay the real timestamps through the limiter offline with the clock injected.

Likely follow-up: “And if it’s rejecting too much?” Look for shared buckets (a proxy’s IP as the key), a capacity smaller than a legitimate batch, and retry storms amplifying the original spike.

Sources

• Lab: system-design/checks/part14_ratelimiter/ (the same token bucket and fixed window in C#, Java, Go and Rust behind an injected clock, plus System.Threading.RateLimiting with manual replenishment); the programs above are run by the series’ code verifiers
• ITU-T Recommendation I.371 (03/2004), Annex A, the Generic Cell Rate Algorithm; RFC 2697 and RFC 2698, token bucket meters; RFC 6585 §4, 429 Too Many Requests; draft-ietf-httpapi-ratelimit-headers-11, RateLimit header fields (Internet-Draft, 23 May 2026)
• .NET: RateLimiter, RateLimitLease, TokenBucketRateLimiter, TokenBucketRateLimiterOptions, PartitionedRateLimiter, rate limiting middleware, TimeProvider
• Java: Guava RateLimiter, java.time.Clock
• Go: golang.org/x/time/rate
• Rust: std::time::Instant; governor (community crate)
• Stripe, Scaling your API with rate limiters; Cloudflare, How we built rate limiting capable of scaling to millions of domains

What to remember

• Decide what you’re protecting, what the unit is, and who the key is, before choosing an algorithm.
• A fixed window allows twice the limit across its boundary. A token bucket caps the burst at its capacity.
• Lazy refill (compute tokens from elapsed time) beats a timer: idle limiters cost nothing.
• Inject a monotonic clock, and clamp negative elapsed time to zero. Then the tests are arithmetic.
• Return when to retry, not just no. In HTTP, that’s 429 plus Retry-After, and check your framework’s default status.
• One lock held for a few arithmetic operations is fine. Partition by key before reaching for lock-free tricks.
• In-process limits are per instance. A global quota needs shared state, approximation, and a decision about what happens when that store is down.

A limiter’s job isn’t to say no. It’s to say no in a way the client can act on.