Requirements Become Numbers: Latency, Percentiles, Availability and Estimation

Fast” and “reliable” can’t be designed for; p99 under 300 ms and 99.9% of requests succeeding can. Learn percentiles, tail latency, why busy servers get slow, availability nines, SLOs, and back-of-the-envelope estimates.

A requirement like “the app should be fast” can’t guide a design, because nobody can check it. “99% of note saves complete in under 300 ms, at 1,700 saves a second” can. It tells you how many servers to plan for, whether a cache is worth it, and when you’ve failed. Turning words into numbers like that is the first real step of a system design.

This post covers the numbers that come up again and again in designs and interviews:

• latency, throughput and percentiles, and why averages mislead;
• why a request that touches many servers is slow more often than any one of them;
• why a busy server gets slow long before it’s full;
• availability “nines”, what they allow, and what they cost;
• how to estimate load and storage on the back of an envelope.

Every calculated number here comes from a script that checks it against its published source, or against a simulation. Numbers quoted from a source are copied into the same script, so any ratio built on them is computed too. The figures are interactive: change the inputs and watch the results move.

Try this first

Two services both have an average response time of 100 ms. Over 1,000 requests:

• Service A answers every request in 90 to 110 ms.
• Service B answers most requests in about 50 ms, but 50 of them take about a second.

Which one would you rather put behind your checkout page? And what single number would have told you the difference?

Then a quick estimate. An app has 10 million daily active users, and each one reads 50 notes a day. Roughly how many reads per second is that? Write down your guess to the nearest power of ten before you reach the estimation section.

Functional and non-functional requirements

Requirements come in two kinds:

• Functional requirements say what the system does: “a user can create a note”, “a user can share a note with another user”, “search returns notes containing a word”.
• Non-functional requirements say how well it does it: how fast, how often it’s available, how much data it keeps and for how long, how consistent reads are, how secure and private it is, and what it costs.

Functional requirements decide which features you build. Non-functional requirements shape the architecture much more than the feature list does. A notes app for one team and a notes app for 10 million people have nearly the same features, and completely different designs.

A non-functional requirement is only useful when you can measure it. Compare:

Vague	Measurable
Fast	99% of note saves complete in under 300 ms (p99 < 300 ms), measured at the load balancer
Scalable	Handles 1,700 saves and 17,000 reads per second at peak
Reliable	99.9% of requests succeed, measured over 30 days
Durable	A save that returned success survives the loss of any one data centre
Private	Note contents are encrypted at rest; deleting an account removes its notes within 30 days

Two details in that table matter:

• “p99 < 300 ms” and “99% of requests in under 300 ms” say the same thing. The next section explains why.
• Every row names where it’s measured, or can be tested. A load balancer is a practical place to measure, but it misses time spent on the user’s network, and often TLS too. Part 1 showed how much that can be, so measure from real clients as well.

The rest of this post is about the numbers in the right-hand column: what they mean, how to measure them, and how to estimate them before anything exists.

Latency and throughput are different things

Latency is how long one request takes. Throughput is how many requests the system completes per unit of time, usually per second.

Latency and throughput are related, but neither one tells you the other. A nightly batch job can move millions of rows an hour while each row waits hours to be processed: high throughput, high latency. One fast server can answer in 5 ms and fall over at 50 requests a second: low latency, low throughput.

Inside a server, a request’s response time has two parts:

• waiting time: how long the request sat in a queue before anything worked on it;
• service time: how long the work itself took once it started.

The user sees more than that, because the network adds its own time on top. Keep waiting time and service time apart. Most of what goes wrong with latency under load is waiting time, and the section on utilisation shows why.

Percentiles: describing latency honestly

Latency isn’t one number. Every request takes a slightly different time, so latency is a distribution, and any single number is a summary of it.

A percentile is the most useful summary. The p99 (99th percentile) latency is the value that 99% of requests were at or below. So “p99 < 300 ms” means at least 99% of requests took under 300 ms, and at most 1 in 100 took longer. The common percentiles are:

• p50, the median: half of requests were faster, half slower. It describes the typical case.
• p90, p95 and p99: the slow end, for 1 in 10, 1 in 20 and 1 in 100 requests.
• p99.9: 1 in 1,000. Services that handle millions of requests a day care about it, because 1 in 1,000 is still thousands of requests.

To compute a percentile exactly, sort the values and pick by rank. With 1,000 requests sorted from fastest to slowest, the p99 is the 990th value. That’s called the “nearest rank” method. Monitoring tools usually estimate percentiles from histograms instead, which is much cheaper, and accurate enough when the histogram’s buckets are narrow near the values you care about.

When the average is the right number

The average isn’t useless. For totals it’s exactly right: total requests, total CPU used, total cost. If you want to know how much a service costs to run, the average load is what you multiply. The average only misleads when you use it to describe what a typical or unlucky user experiences.

Why the average lies about latency

Here are Services A and B from the start, generated by the checks script. Step to step 4, then switch between the two services.

The same comparison in words:

	Mean	p50	p95	p99
Service A	100 ms	100 ms	109 ms	110 ms
Service B	100 ms	53 ms	62 ms	1,070 ms

• Service A: every user gets about the same experience.
• Service B: 95% of requests take between 42 and 62 ms, about half of A’s time, and 5% take about a second.

The means are identical, so a dashboard that shows only the average can’t tell these services apart. The median and p95 make B look better than A, because B’s slow requests are exactly the slowest 5%. Only p99 shows the problem.

Google’s Site Reliability Engineering book makes the same point about real systems: “Most metrics are better thought of as distributions rather than averages.” In its example, “a typical request is served in about 50 ms, 5% of requests are 20 times slower”, and a graph of the average showed no change at all.

Why the slow 1% matters more than it sounds

“Only 1 request in 100 is slow” sounds harmless. It isn’t:

1. Users make many requests. A page that loads 50 resources, each with its own 1-in-100 chance of being slow, hits at least one slow request on about 4 page views in 10. The next section explains the arithmetic.
2. The slow requests are often the important ones. The customer with the biggest basket, or the most data, often has the slowest request.

You can’t average percentiles

Here’s a mistake that looks reasonable: each server reports its own p99, and a dashboard shows the average of them. These two servers come from the checks script:

	Requests	p99
A busy, healthy server	9,000	30 ms
A quiet server with a problem (20 slow requests)	1,000	781 ms
The average of the two p99s		405 ms
The real p99 of all 10,000 requests		77 ms
The real p99.9 of all 10,000 requests		781 ms

The average of the two p99s is five times the real p99. It gives a server with a tenth of the traffic the same weight as the busy one. Whether the average comes out too high or too low depends on how traffic and slow requests are spread, so there’s no correction you can apply.

The table also shows why one correct number isn’t enough. The real p99, 77 ms, looks healthy. But one server is sick, and the merged p99.9 shows it.

The right way to combine percentiles is to keep each server’s distribution, as a histogram, and merge the histograms before reading a percentile. Merging means adding up the counts in matching buckets, so every server must use the same bucket boundaries. Mergeable sketches, such as HDR histograms, t-digest and DDSketch, exist for exactly this. The same rule applies over time: the p99 of a day isn’t the average of 24 hourly p99s. Then look at the result per server too, not only merged.

Tail latency at scale

A single request from a user often isn’t served by one server. A search query, a feed or a product page can fan out to tens or hundreds of backend servers in parallel, and the response is only ready when all of them have answered.

Suppose each backend server is slow for only 1 request in 100. How often is the user’s request slow? It’s slow whenever at least one server is slow. The chance that all N servers are fast is 0.99^N, so:

P(the user’s request is slow) = 1 − 0.99^N

Move the slider and watch what happens as N grows.

The numbers, for servers that are each slow 1 time in 100:

Servers per request	Chance the request is slow
1	1.0%
10	9.6%
20	18.2%
50	39.5%
100	63.4%
200	86.6%

From 69 servers up, more than half of all user requests are slow. So at about 70 servers, one server’s p99 has become the whole system’s median.

This is the example in Jeff Dean and Luiz André Barroso’s paper “The Tail at Scale” (Communications of the ACM, 2013). With 100 servers, they write, “63% of user requests will take more than one second”. They add that even when only 1 in 10,000 requests is slow at a single server, “a service with 2,000 such servers will see almost one in five user requests taking more than one second”. The formula gives 18.1%.

The paper also reports real measurements from a Google service that fans out to many leaf servers:

• One random leaf request: 10 ms at the 99th percentile.
• All leaf requests finished: 140 ms at the 99th percentile.
• 95% of leaf requests finished: 70 ms.

So, in the paper’s words, “waiting for the slowest 5% of the requests to complete is responsible for half of the total 99%-percentile latency.”

Explain it like I’m ten

Your class is going on a trip, and the bus leaves only when every child is on board. Each child is late 1 day in 100.

If you’re the only child, the bus is late 1 day in 100. With a class of 100 children, it’s very likely that at least one of them is late on any given day. The bus would be late on about 63 days out of 100, even though each child is almost always on time.

The precise version

A request that waits for N sub-requests is as slow as the slowest of them. If each sub-request exceeds a latency t with probability p, independently of the others, the whole request exceeds t with probability 1 − (1 − p)^N. As N grows, the whole request’s typical latency moves towards the sub-requests’ tail.

Where the analogy breaks: children being late on the same day aren’t always independent. A storm makes many of them late together. Servers are similar, because a shared network switch or a garbage-collection pause can slow many at once. The formula assumes independence, so real systems can do better or worse than it, depending on how their slow moments line up.

What to do about it

You can’t make every server fast every time, so large systems design around the tail. The paper describes several techniques, including these:

• Hedged requests: send the request to one replica. If it hasn’t answered within a short delay, send it to a second replica too, use whichever answers first, and cancel the other. The paper suggests waiting for the 95th-percentile latency before sending the second copy, which limits the extra load. In one measurement, hedging after 10 ms “reduces the 99.9th-percentile latency for retrieving all 1,000 values from 1,800ms to 74ms while sending just 2% more requests.”
• Tied requests: send the request to two servers, each told about the other. When one starts working on it, it tells the other to cancel. Both can start in the brief moment before the cancellation arrives, so the paper suggests a small delay, “1ms or less in modern data-center networks”, before sending the second copy.
• Manage background work: throttle it, break it into smaller pieces, and run it when load is low. For large fan-out services, the paper suggests sometimes running background work on all machines at the same moment, so that only the requests during that short burst are slowed, instead of a few machines always slowing everyone.

Two more options follow from the arithmetic, rather than from the paper:

• Fan out to fewer servers, so each user request has fewer chances to hit a slow one.
• Return a partial result when a few servers are late, if the product can accept one.

Hedging and tied requests send some requests twice, so they’re only safe for idempotent requests, which Part 1 explained. Part 33 covers these resilience patterns in more depth.

Utilisation: why busy servers get slow

A server at 90% busy isn’t a little slower than one at 80%. In the simplest model, it’s twice as slow. That steep curve is why capacity plans leave headroom.

The simplest model of a server is a queue in front of one worker:

• Requests arrive at random, at an average rate λ (lambda) per second.
• The worker serves them one at a time, first come first served, at an average rate μ (mu) per second. So each request takes 1/μ seconds of work on average.
• Utilisation, ρ (rho), is the fraction of time the worker is busy: ρ = λ / μ.

When arrivals are random (a Poisson process) and service times are random (exponentially distributed), this is called an M/M/1 queue. For it, the average time a request spends in the system, waiting plus being served, is:

W = 1 / (μ − λ) = service time / (1 − ρ)

Look at the 1 − ρ at the bottom. As utilisation approaches 100%, it approaches zero, so the time in the system grows without limit. Move the slider to see a server with a 10 ms average service time.

The same numbers as a table:

Utilisation	Arrivals per second	Mean time in system	p99 time in system	Mean requests waiting
50%	50	20 ms	92 ms	0.5
70%	70	33 ms	154 ms	1.6
80%	80	50 ms	230 ms	3.2
90%	90	100 ms	461 ms	8.1
95%	95	200 ms	921 ms	18.1

Going from 80% to 90% busy adds only 10 requests a second, but it doubles both the mean and the p99 latency. Going from 90% to 95% doubles them again. The work per request never changed. Every extra millisecond is spent waiting in the queue.

In this model, time in the system follows an exponential distribution, so its p99 is ln(100) × W, about 4.6 times the mean. The checks script simulates the queue with 400,000 requests at 50%, 80% and 90% utilisation, averaged over three random seeds. The simulated means and p99s match the formulas within a few percent.

Explain it like I’m ten

A shop has one cashier, and each customer takes about a minute.

If a customer arrives every two minutes on average, the cashier is busy half the time, and you rarely wait long. But customers don’t arrive evenly. Sometimes three come at once, and a small line forms. The cashier works through that line using their spare time, the minutes when nobody new arrives.

When the cashier is busy nine minutes out of ten, there’s very little spare time. A small bunch of customers takes a long time to clear, and the next bunch often arrives before it has. So the line stays long, even though, on average, the cashier can keep up.

The precise version

A queue forms when arrivals briefly outpace the server. The server clears the backlog with its spare capacity, the fraction 1 − ρ by which it’s faster than the average arrival rate. A burst of work takes about 1/(1 − ρ) times as long to clear as it would on an idle server. So halving the spare capacity, from 20% to 10%, doubles the average wait. For M/M/1, the average number of requests waiting (not counting the one being served) is ρ² / (1 − ρ), which grows without limit as ρ approaches 1.

Where the analogy breaks: real servers differ from M/M/1 in ways that change the exact numbers:

• More variable service times make queues worse. A few very slow requests make everyone behind them wait.
• Several workers sharing one queue make them better. Here’s the mean time in the system at 90% utilisation per worker, with a 10 ms service time, from the M/M/c (Erlang C) formula. The script also simulates the 4-worker case, and gets 30.0 ms.

Workers sharing one queue	Mean time in system
1	100 ms
4	29.7 ms
8	18.8 ms
32	11.4 ms

• Separate queues don’t help. Workers that each have their own queue behave like separate M/M/1 servers, even if they’re on one machine.

So a 32-core server whose threads share one work queue can run far busier than one single-threaded server before latency rises. What carries over to every case is the shape: flat for a long time, then steep. Where the steep part starts depends on the system, and a load test finds it.

What this means for design

• Don’t plan to run near the steep part. Load-test the service, find the utilisation where the p99 objective starts to break, and keep enough margin below it to absorb a traffic spike or the loss of a server.
• Remember what losing a server does. Ten servers at 90% utilisation that lose one are pushed to 100%, and the queue grows without limit.
• Watch queues and waiting time, not only CPU. A service can show modest average CPU and still queue requests during bursts. CPU also isn’t the same as utilisation when requests wait on locks, disks or other services.
• Reduce variability. Separate slow requests from fast ones, for example with a separate pool or a separate queue, so a few big requests don’t make everyone wait.
• Shed load when a queue gets long. A fast “try again later” is better than a slow timeout. Part 33 covers load shedding.

Availability, and what the nines cost

Availability is the fraction of the time, or of requests, that the system works. It’s quoted in “nines”: 99.9% is “three nines”.

Availability can be measured in two ways, and Google’s SRE book describes both:

• Time-based: the fraction of time the service was up. “Up” needs a definition, such as “health checks pass from three regions”.
• Aggregate (request-based): the fraction of requests that succeeded. For a large, distributed service that’s rarely completely up or completely down, this is usually the better measure. The book’s example: at 99.99%, a service that serves 2.5 million requests in a day can fail up to 250 of them and still meet its target.

Either way, you have to decide what counts as a failure. A common starting point:

• Server errors (HTTP 5xx) count against availability.
• Client errors (4xx) usually don’t, because they’re the client’s mistake. A sudden jump in them may still mean something broke.
• A request slower than its latency objective can count as failed, if the SLI says so.
• Planned maintenance counts unless the agreement with users explicitly excludes it. The table below assumes none.

Here’s how much downtime each level allows. Move the slider.

Availability	Per year	Per 30 days	Per day
99%	3.65 days	7.20 hours	14.40 minutes
99.5%	1.83 days	3.60 hours	7.20 minutes
99.9%	8.76 hours	43.20 minutes	1.44 minutes
99.95%	4.38 hours	21.60 minutes	43.20 seconds
99.99%	52.56 minutes	4.32 minutes	8.64 seconds
99.999%	5.26 minutes	25.92 seconds	0.86 seconds

These use a 365-day year and a 30-day month, which is the convention the SRE book’s availability table implies. The checks script compares every cell the book prints with the exact value. They all agree except one, where the book rounds 0.864 seconds up to 0.87.

Each extra nine cuts the allowed downtime by ten times, and it can cost far more than ten times as much to achieve. The SRE book says an incremental improvement in reliability “may cost 100x more than the previous increment.” At five nines, a month allows 26 seconds of downtime. A person can’t notice and fix an outage in 26 seconds, so recovery has to be automatic. That means redundancy, automated failover, and careful deployments.

Many systems don’t need that. Choosing the target is a business decision, and it drives much of the design’s cost.

Dependencies multiply

A request that needs several services to work is only available when all of them are. If their failures are independent, their availabilities multiply. Move the slider to chain services that are each 99.9% available.

Services the request needs	Availability, if failures are independent	Downtime per year
1	99.90%	8.8 hours
3	99.70%	26.3 hours
5	99.50%	43.7 hours
10	99.00%	87.2 hours
20	98.02%	173.5 hours

Two limits follow:

• A request can never be more available than its least available hard dependency. Even if all five dependencies always failed at exactly the same moments, the request would be down whenever they were, so 99.9% is the ceiling.
• With independent failures, expect about the product. For five 99.9% dependencies, that’s 99.5%.

So a 99.99% target on top of five 99.9% dependencies isn’t achievable by trying harder. It needs a design change:

• Fewer hard dependencies.
• Degraded answers when a dependency fails.
• Retries against short, independent failures. If each attempt fails independently 0.1% of the time, two attempts both fail only 0.0001% of the time. Retries don’t help against a dependency that’s actually down, and careless retries can overload it. Part 23 covers how to retry safely.

Redundancy works the other way. If a request needs any one of several independent replicas, it only fails when all of them fail. Two replicas that are each 99% available give 1 − 0.01² = 99.99%.

That calculation makes two assumptions:

• The system notices a failed replica and moves traffic instantly. Detection and failover take time, and that time is downtime too.
• The replicas fail independently. Real replicas share things: a power supply, a network, a region, a configuration file, a deployment. A bad configuration pushed to both replicas takes both down together, and the 99.99% was never real.

For a chain, failures that happen together are less damaging, because the outages overlap. For redundancy, they’re the main risk. That’s why Part 38 puts replicas in separate failure domains.

Durability isn’t availability

Durability is whether data, once acknowledged as saved, stays saved. Availability is whether you can use the system right now. They’re separate properties:

• A system can be unavailable for an hour and lose nothing.
• A system can be available and quietly lose data.

They need separate requirements, because they lead to different mechanisms. Durability comes from replicating writes before acknowledging them, checksums, and backups you’ve actually restored. Availability comes from redundancy and fast failover.

The mechanisms can pull against each other. Waiting for a write to be copied to another data centre protects it, but if that data centre is unreachable, the write can’t complete.

SLIs, SLOs and SLAs

These three terms come up in every reliability conversation, and the SRE book defines them:

• SLI, service level indicator: “a carefully defined quantitative measure of some aspect of the level of service that is provided”. For example: the fraction of note saves that complete successfully in under 300 ms, measured at the load balancer.
• SLO, service level objective: “a target value or range of values for a service level that is measured by an SLI”. For example: that SLI is at least 99.9%, over a rolling 30 days.
• SLA, service level agreement: “an explicit or implicit contract with your users that includes consequences of meeting (or missing) the SLOs they contain”. For example: if availability falls below 99.5% in a month, customers receive a 10% credit.

The book offers a simple test. Ask “what happens if the SLOs aren’t met?” If there’s no explicit consequence, “you are almost certainly looking at an SLO.”

An SLO also gives you an error budget, the failures you’re allowed. A 99.9% SLO over 30 days allows 0.1% of requests to fail. The SRE book describes using it to manage releases. As long as budget remains, releases continue. If the budget is spent, “releases are temporarily halted” while the team works on reliability. Part 35 covers SLOs, error budgets and alerting in depth.

Two practical points about SLIs:

• Measure close to the user. A load balancer is a good practical place, but it misses the user’s DNS lookups and network. Part 1 showed those can be most of the time.
• Measure durations with a monotonic clock. A wall clock can jump when the system synchronises its time, which can make a measured duration wrong, or even negative. Each platform has a clock meant for measuring elapsed time:

Platform	Clock	What its documentation says
.NET	Stopwatch	It uses a high-resolution performance counter when one exists, which Stopwatch.IsHighResolution reports
Java	System.nanoTime()	“can only be used to measure elapsed time and is not related to any other notion of system or wall-clock time”
Go	the monotonic reading inside time.Now()	“the wall clock is for telling time and the monotonic clock is for measuring time”. Rounding, truncating or converting a time drops that reading
Rust	std::time::Instant	“A measurement of a monotonically nondecreasing clock”. It can still jump forwards, for example around a system suspend

Back-of-the-envelope estimation

Before designing anything, estimate the load. The goal isn’t precision. It’s the right order of magnitude, because a design for 10 requests a second and a design for 10,000 are different designs, while 10,000 and 12,000 usually aren’t.

The numbers to remember

• A day has 86,400 seconds. You can round it to 100,000 to estimate quickly, but the rounding makes your answer about 14% low.
• So 1 million requests a day is about 11.6 a second on average.
• Peak is higher than average, because traffic follows the time of day. If the busiest hour carries 10% of a day’s traffic, the peak rate is 0.10 × 24 = 2.4 times the daily average. If you have no data yet, state an assumption, and measure it once real traffic exists.
• Storage units: 1 KB = 10³ bytes, 1 MB = 10⁶, 1 GB = 10⁹, 1 TB = 10¹² and 1 PB = 10¹⁵. Memory and some tools use powers of two instead, where 1 KiB = 1,024 bytes. For estimates, the difference doesn’t matter.

A worked example: a notes app

Here’s the method, applied to the notes app from the start. Every input is an assumption, and a good estimate names each one:

Assumption	Value
Daily active users	10 million
Notes saved per user per day	5
Notes read per user per day	50
Peak traffic compared with the daily average	3 times
Size of one note, with its metadata	2 KB
Copies of the data kept	3
Reads one server handles at the target utilisation (from a load test)	1,000 per second

And the results, computed by the checks script:

Estimate	Calculation	Result
Writes per second, average	10M × 5 ÷ 86,400	579
Writes per second, peak	× 3	about 1,700
Reads per second, average	10M × 50 ÷ 86,400	5,787
Reads per second, peak	× 3	about 17,000
Read-to-write ratio	50 ÷ 5	10 to 1
New data per day	10M × 5 × 2 KB	100 GB
New data per year, one copy	× 365	36.5 TB
New data per year, three copies	× 3	109.5 TB
Servers for the peak read load	17,361 ÷ 1,000, rounded up, plus 2 spare	20

Now read what the estimate says about the design:

• Reads outnumber writes 10 to 1, at about 17,000 a second at peak. That makes it worth asking early whether every read should reach the primary database, or whether read replicas and a cache should take most of them. Parts 20 and 22 cover both.
• One copy of the data grows by about 37 TB a year. At some point that won’t fit on one machine, which points towards partitioning (Part 21). It also raises the question of whether old notes need to be as fast to read as new ones.
• The server count depends on the load test. The 1,000 reads per server is an input you measure, not a fact about servers. Whether one machine can take a given load depends on its hardware, the queries and the data. The estimate tells you which questions to test.

If you guessed thousands of reads a second at the start, you were close. It’s about 6,000 on average and 17,000 at peak, so the nearest power of ten is 10⁴.

Latency numbers, as orders of magnitude

Peter Norvig’s essay “Teach Yourself Programming in Ten Years” includes a table of approximate timings for a typical PC. Here are some of its entries:

Operation	Approximate time
Fetch from L1 cache	0.5 ns
Fetch from main memory	100 ns
Send 2 KB over a 1 Gbps network	20,000 ns (0.02 ms)
Read 1 MB sequentially from memory	250,000 ns (0.25 ms)
Fetch from a new disk location (seek)	8,000,000 ns (8 ms)
Read 1 MB sequentially from disk	20,000,000 ns (20 ms)
Send a packet from the US to Europe and back	150 ms

The disk rows describe spinning disks, which have to move a head to seek. SSDs don’t, so don’t quote those values for modern storage. What still holds is the gap between the layers. In this table, fetching from memory is 200 times faster than sending 2 KB across a gigabit local network. That in turn is 7,500 times faster than a round trip across the Atlantic. Those gaps are why caches exist, and why the round trips in Part 1 dominated. When you need a real number for your hardware, measure it.

Trade-offs

Higher availability versus cost and speed of change. Every nine costs redundancy, automation and slower, more careful deployments. Pick the lowest target your users will accept, and spend the rest of the budget on features.

Utilisation versus latency. Running servers busier saves money and costs tail latency. The steep part of the curve is where the savings stop being worth it.

Throughput versus latency. Batching work, such as writing ten rows at once, raises throughput, and makes each item wait for its batch. Many systems let you tune both the batch size and the maximum wait.

Fan-out versus tail latency. Splitting work across more servers makes each server’s job smaller, and makes the request more likely to hit a slow moment. Hedging helps, at the cost of extra load.

Measuring precisely versus measuring cheaply. Keeping every latency value is expensive. Histograms are cheap and mergeable, and accurate enough when their buckets are narrow near the values you care about.

Common mistakes

Setting objectives on the average. The average hides slow requests, and users notice slow requests. Use percentiles for latency objectives.

Averaging percentiles across servers or time windows. That isn’t a valid way to combine them, as the table above showed. Merge histograms, then read the percentile, and still look per server.

Measuring latency only inside the server. Waiting in a queue before your code runs, and all the network time, stay invisible. Measure at the edge and from real clients too.

Planning capacity for the average day. The average hides the peak, and the peak is when systems break. State the peak factor, plan for losing a server at peak, and test at peak load.

Promising an availability the dependencies can’t support. Check the least available hard dependency, and multiply the availabilities for the expected value. If the target is above those numbers, change the design or lower the target.

Assuming replicas fail independently. Shared configuration, deployments and infrastructure make them fail together. Put replicas in separate failure domains, and roll out changes gradually.

Measuring durations with the wall clock. DateTime.Now, System.currentTimeMillis() and similar clocks can jump. Use the monotonic clock each platform provides.

Interview questions

These questions on latency, availability and estimation come up in system design interviews at every level. Try answering each one out loud before you open the answer.

1. What does “p99 latency is 250 ms” mean, and why do we use percentiles instead of the average?

Show a strong answer

It means at least 99% of requests completed in 250 ms or less, and at most 1% took longer.

We use percentiles because latency is a distribution, and the average hides its shape. Two services can share a 100 ms average while one has a p99 of 110 ms and the other a p99 of over a second. The tail matters because:

• users make many requests, and pages load many resources, so most users hit the tail often;
• requests that fan out to many servers inherit the servers’ tails;
• the slow requests are often the heaviest and most valuable ones.

A strong answer mentions p50 for the typical case, p99 or p99.9 for the tail, and that percentiles must be computed from merged distributions, not averaged. It also knows when the average is right: for totals like cost and capacity.

Likely follow-up: “How do you compute p99 across 50 servers?” Merge their histograms, which need the same bucket boundaries, then read the percentile.

2. A request fans out to 100 backend servers. Each one has a p99 of 1 second. What’s the chance a user request takes more than a second?

Show a strong answer

A p99 of 1 second means each server is slower than a second about 1% of the time. The user’s request is slow if any of the 100 is slow. Assuming independence, the chance that all are fast is 0.99¹⁰⁰ ≈ 0.366. So the chance that at least one is slow is about 63%. This is the example from Dean and Barroso’s “The Tail at Scale”.

How to reduce it:

• Hedged requests: send a second copy to another replica after a short delay, such as the p95 latency, and take the first answer.
• Tied requests, which cancel the duplicate once one server starts working on it.
• Fan out to fewer servers.
• Reduce the variability at each server: manage background work, and keep queues short.
• Return partial results when a few servers are late, if the product allows it.

Likely follow-up: “What’s the cost of hedging, and when is it unsafe?” It adds some load, only a few percent when the delay is chosen well. It’s only safe for idempotent requests.

3. Why shouldn’t you run a latency-sensitive service at 90% CPU?

Show a strong answer

Because waiting time grows much faster than load. In the simplest queueing model, M/M/1, the mean time in the system is the service time divided by (1 − utilisation):

• at 50%, 2 times the service time;
• at 80%, 5 times;
• at 90%, 10 times;
• at 95%, 20 times.

Going from 80% to 90% doubles the mean, and in that model the p99 doubles too.

A strong answer adds three things:

• Ask “90% of what?” Many workers sharing one queue behave much better than one worker. With 32 workers at 90% each, the M/M/c mean is about 1.1 times the service time, not 10 times. CPU also isn’t the same as utilisation when requests wait on locks, disks or other services.
• Leave room for failure. Ten servers at 90% that lose one are pushed to 100%, and their queues grow without limit.
• Set the target with a load test. Find where the p99 objective breaks, keep a margin below it, and watch queue lengths as well as CPU.

Likely follow-up: “What would you do when load approaches the limit?” Add capacity ahead of it, and shed load or degrade gracefully when a queue grows (Part 33).

4. What’s the difference between an SLI, an SLO and an SLA? Give an example of each.

Show a strong answer

• SLI: the measurement. “The fraction of checkout requests that return a non-5xx response in under 500 ms, measured at the load balancer.”
• SLO: the target for that measurement. “That SLI is at least 99.9%, over a rolling 30 days.” SLOs are often internal, but some teams publish them.
• SLA: the contract with consequences. “If monthly availability is below 99.5%, customers receive a 10% service credit.”

Set the SLA looser than the SLO, so you find out you’re in trouble before you owe anyone money. The SLO gives an error budget, here 0.1% of requests, which decides how fast the team can keep releasing changes.

Likely follow-up: “How would you alert on an SLO?” Alert on how fast the error budget is being used up, which Part 35 covers.

5. Your service depends on five other services, each 99.9% available. Can you offer 99.99%?

Show a strong answer

Not if all five are hard dependencies called on every request:

• The ceiling is 99.9%, the availability of the weakest dependency, even if they all failed at exactly the same moments.
• With independent failures, expect about 0.999⁵ ≈ 99.5%. That’s about 44 hours of downtime a year, against about 53 minutes for 99.99%.

To get higher, change the design:

• Make dependencies soft: return a degraded but useful response when one fails, such as a page without recommendations.
• Cache or replicate what you need from them, so a short outage doesn’t reach users.
• Retry short, independent failures, with limits, so retries don’t overload a struggling dependency.
• Make calls asynchronous where the user doesn’t need the result immediately, for example with a queue.
• Add redundancy to the dependencies themselves, in separate failure domains.

Likely follow-up: “Two replicas at 99% each give 99.99%. Is that real?” Only if they fail independently and failover is instant. Shared configuration, deployments and infrastructure usually break the first assumption, and detection time breaks the second.

6. Estimate the traffic and storage for a service with 50 million daily users who each upload 2 photos a day, averaging 500 KB.

Show a strong answer

Say the assumptions out loud, then calculate. The answers below come from the checks script:

• Uploads: 50M × 2 = 100M a day. Divided by 86,400 seconds, that’s about 1,160 a second on average, or about 3,500 at an assumed 3 times peak. Each upload is also a metadata write to a database, at the same rate.
• Storage: 100M × 500 KB = 50 TB a day, which is about 18 PB a year for one copy. That’s before replicas, and before the smaller resized versions most photo services also store.
• Reads: assume each user views 100 photos a day. That’s 5 billion views a day, about 58,000 a second on average. Reads dominate.
• Bandwidth: uploads average about 4.6 Gbps inbound. If every view served the full 500 KB image, outbound traffic would average about 231 Gbps, and about 694 Gbps at peak. That number is why real services serve smaller versions for thumbnails and feeds, from a CDN.

What it tells you:

• Store the photos in object storage and serve them through a CDN, with only metadata in a database.
• Caching at the edge matters most, because reads outnumber uploads 50 to 1 and carry most of the bandwidth.
• At 18 PB a year, the cost levers are deduplication, smaller stored versions, and moving old photos to cheaper storage tiers. Photos are already compressed, so compressing them again saves little.

A strong answer rounds sensibly, names every assumption, and turns the numbers into design consequences.

Likely follow-up: “Where are you most uncertain?” The peak factor, the view pattern, and how many resized versions are stored.

7. Why can’t you average the p99s reported by each of your servers?

Show a strong answer

A percentile describes a whole distribution, and percentiles don’t combine by averaging. An average gives a server with a tenth of the traffic the same weight as a busy one, and it ignores how the values are spread.

In the example in this post, a busy server with a p99 of 30 ms and a quiet one with a p99 of 781 ms average to 405 ms. The real p99 across all requests was 77 ms. Depending on the data, the average can be too high or too low.

The fix is to record latency as histograms on each server, using the same bucket boundaries everywhere, or as a mergeable sketch. Merge them by adding the counts, then read the percentile from the merged result. The same applies across time windows. Keep the per-server view too: in the example, the healthy-looking merged p99 hid a server with 2% of its requests taking most of a second.

Likely follow-up: “What’s the trade-off with histograms?” The bucket boundaries limit accuracy, so make them narrow near your SLO threshold.

8. What’s the difference between availability and durability?

Show a strong answer

• Availability: can you use the system right now? It’s measured as uptime, or as the fraction of successful requests.
• Durability: once the system says your data is saved, does it stay saved? It’s measured as how likely an acknowledged write is to be lost.

They’re separate properties. A database can be offline for an hour and lose nothing: low availability, high durability. A cache-only store can be up all the time and lose everything on a restart: high availability, low durability.

They need different mechanisms, and those mechanisms can pull against each other:

• Durability comes from replicating writes before acknowledging them, checksums, and backups you’ve actually restored.
• Availability comes from redundancy, health checks and failover.
• The conflict: waiting for a write to reach another data centre protects it, but makes the write fail if that data centre can’t be reached.

Likely follow-up: “When would you trade one for the other?” For example, acknowledging a write before it’s replicated keeps writes fast and available, and risks losing the most recent ones if a machine fails.

Sources

• Jeffrey Dean and Luiz André Barroso, The Tail at Scale, Communications of the ACM 56(2), February 2013 (fan-out, measurements, hedged and tied requests, background work)
• Betsy Beyer, Chris Jones, Jennifer Petoff and Niall Richard Murphy (eds.), Site Reliability Engineering, O’Reilly, 2016: Service Level Objectives, Embracing Risk, and Appendix A, Availability Table
• Peter Norvig, Teach Yourself Programming in Ten Years (approximate timings table)
• Queueing results for M/M/1 and M/M/c (Erlang C), as derived in standard queueing theory texts such as Leonard Kleinrock, Queueing Systems, Volume 1: Theory, Wiley, 1975. We checked them against simulations rather than taking them on trust.
• Microsoft, Stopwatch class; Oracle, System.nanoTime, Java SE 25; Go, time package: Monotonic Clocks; Rust, std::time::Instant
• Checks: system-design/checks/part02_numbers.py. It compares every cell of the SRE availability table and the Tail at Scale examples, simulates the M/M/1 and M/M/4 queues, computes every estimate in the post, and generates every figure’s data.

What to remember

• A requirement you can’t measure can’t guide a design. Write non-functional requirements as numbers, and say where each one is measured.
• Latency is a distribution. Use percentiles, not the average, and merge histograms rather than averaging percentiles.
• A request that waits for many servers is slow whenever any of them is: 1 − (1 − p)^N. At 100 servers, a 1-in-100 tail becomes 63% of requests.
• For one queue and one worker, time in the system grows like 1 / (1 − utilisation). Going from 80% to 90% busy doubles it. More workers sharing a queue move the steep part later, and a load test tells you where it is.
• Each nine of availability allows ten times less downtime. A request can’t be more available than its weakest hard dependency, and redundancy only helps when failures are independent.
• An SLI is the measurement, an SLO the target, and an SLA the contract with consequences.
• Estimate before you design: 86,400 seconds a day, name every assumption, and look for the power of ten.

Write every requirement as a number you can measure, and design for the slow requests, not the average.