Why Distributed Systems Are Hard: Partial Failure, Timeouts and Retries

A remote call can succeed, fail, or tell you nothing. Measured on real HTTP: lost replies charge customers more than once, three of four default clients wait 150 s or more, and retries multiply 243 times over five layers, then keep a simulated server down.

Inside one program, a function call either returns or throws. You always find out which.

Across a network there’s a third outcome: nothing. No answer, no error, just time passing. You don’t know if the other side got your request, did the work and lost the reply, or never heard from you at all. Almost everything that makes distributed systems hard follows from that one fact.

This part measures what that third outcome does. A payment server loses a fifth of its replies. Default HTTP clients in four languages wait on a server that never answers. Retries run through a chain of five real services. And a simulated server stalls for two seconds and, with the wrong retry policy, never recovers.

Try this first

Three questions. Write down an answer to each before you scroll.

1. Your code sends a payment request, and the connection drops before the reply arrives. Your code retries. Of 1,000 payments, where a fifth of the replies go missing, how many customers get charged twice?
2. A request passes through five layers: your client and four services. Each one makes up to three attempts at every call. The database at the bottom is down. How many requests reach the database for one click?
3. You call a server that accepts the connection and then never answers. With no timeout configured, how long does your HTTP client wait?

A call that neither succeeds nor fails

In 1994, four engineers at Sun Labs wrote the argument that still frames the subject:

Partial failure is a central reality of distributed computing. Both the local and the distributed world contain components that are subject to periodic failure. In the case of local computing, such failures are either total, affecting all of the entities that are working together in an application, or detectable by some central resource allocator (such as the operating system on the local machine). This is not the case in distributed computing, where one component (machine, network link) can fail while the others continue.

And one sentence further on, the part that matters most for the code you write:

In a distributed system, the failure of a network link is indistinguishable from the failure of a processor on the other side of that link.

Leslie Lamport put it more memorably in an email to his colleagues at DEC’s research lab in May 1987, which is where the famous line actually comes from. It isn’t from a paper:

A distributed system is one in which the failure of a computer you didn’t even know existed can render your own computer unusable.

The lab: a payment server that loses its replies

The lab runs a payment server on localhost. It charges the payment, and then, for a random 20% of attempts, closes the connection instead of replying. To the client that looks exactly like a network failure. The client makes up to five attempts per payment.

	Without idempotency keys	With idempotency keys
Payments	1,000	1,000
Attempts that reached the server	1,244	1,244
Charges actually made	1,244	1,000
Payments charged more than once	201	0

Two hundred and one customers charged twice or more, which is about the 20% whose first reply was lost. Every retry was reasonable, since the client had no answer. And every retry was wrong, because the charge had already happened.

The fix is in the right-hand column. The client sends an idempotency key, one per payment and the same on every retry, and the server remembers which keys it has already processed. A repeat gets the stored result instead of a second charge. Stripe’s API works this way, and its documentation says what a missing answer means:

When intermittent problems occur, clients are usually left in a state where they don’t know whether or not the server received the request. To get a definitive answer, they should retry such requests with the same idempotency keys and the same parameters until they’re able to receive a result from the server.

Stripe keeps keys for about 24 hours, so a retry has to happen inside that window.

Retries nobody wrote

Look at the right-hand column again. The server received 1,244 attempts. But the program, counting its own calls, made only 1,043.

The other 201 were sent by Go’s HTTP library, without being asked. Go’s http.Transport retries a request by itself after a network error on a connection it had used before, if it considers the request idempotent. And a POST counts as idempotent if it carries an idempotency key:

Transport only retries a request upon encountering a network error if the connection has already been used successfully and if the request is idempotent and either has no body or has its [Request.GetBody] defined. HTTP requests are considered idempotent if they have HTTP methods GET, HEAD, OPTIONS, or TRACE; or if their [Header] map contains an “Idempotency-Key” or “X-Idempotency-Key” entry.

The conditions are narrow. The connection must have been used before, and the failure must come before any part of a reply. In this lab, with one request at a time, the retry had to go out on a fresh connection, because the server had just closed the only one. So Go retried each lost reply at most once. And it never retries a request that simply times out. That’s why the program saw only 43 of the 244 lost replies. By the rule above, those were losses on connections Go had just opened, such as the one it opened for a retry.

HTTP’s own specification allows this. A client “can repeat a POST request automatically if it knows (through design or configuration) that the request is safe for that resource”, and an idempotency key is that knowledge, if the server honours it. What the specification calls “a riskier approach” is guessing without it.

But the retry is only safe if the server honours the key. Go retries whenever the header is present. If this server had ignored keys, those 201 hidden retries would have been 201 double charges that the program never saw. The number of attempts your code sees isn’t the number your server sees. There can be a retry layer you never wrote.

.NET’s standard resilience handler goes further by default. Microsoft’s documentation says so directly:

By default, the standard resilience handler is configured to make retries for all HTTP methods. For some applications, such behavior could be undesirable or even harmful. For example, if a POST request inserts a new record to a database, then making retries for such a request could lead to data duplication.

It retries on connection errors too, which is the lost-reply case above.

The lesson isn’t “never retry”. It’s that every retry, including the ones you didn’t write, needs the operation to be safe to repeat.

You can’t always find out

A cleverer protocol can’t remove the doubt. Two classic results say why.

The first is usually told as a story about two generals. Jim Gray’s 1978 notes on database systems tell it this way. Two generals must attack together or not at all. They can only communicate by messengers, and any messenger might get lost. Gray gives a four-line proof that no fixed-length exchange of messages can guarantee they agree:

There is a simple proof that: no fixed length protocol exists: Let P be the shortest such protocol. Suppose the last messenger in P gets lost. Then either this messenger is useless or one of the generals doesn’t get a needed message. By the minimality of P, the last message is not useless so one of the general doesn’t march if the last message is lost. This contradiction proves that no such protocol P exists.

Gray cites no one for the problem, and Lamport describes it as already known when he borrowed the generals for his own work. Who first posed it is less certain than the story usually makes it.

That’s your payment. The server charged, and its reply was the messenger that got lost. No number of acknowledgements closes the gap, because the last acknowledgement can always be the one that’s lost.

The second result is Fischer, Lynch and Paterson’s, from 1985, usually called FLP:

In this paper, we show the surprising result that no completely asynchronous consensus protocol can tolerate even a single unannounced process death.

Read what “completely asynchronous” means in their model, because it’s the whole trick:

Finally, we do not postulate the ability to detect the death of a process, so it is impossible for one process to tell whether another has died (stopped entirely) or is just running very slowly.

A crashed machine and a slow one look the same from outside. FLP doesn’t say consensus is impossible in practice. Part 26 is about a protocol that does it every day. It says no algorithm can guarantee to finish if it can’t tell slow from dead. Real systems escape by adding what the model leaves out: timeouts, which let you guess that something is dead. The rest of this part is about what those guesses cost.

The default is to wait

A timeout is the only way a client ever stops waiting for a server that has stopped talking. So what happens when you don’t set one?

The lab started a server that accepts the connection, reads the request and never answers. Then it pointed each language’s standard HTTP client at it, with nothing configured.

Client, nothing configured	What happened
Go net/http, http.DefaultClient	still waiting when the lab stopped it at 150 s
Java java.net.http.HttpClient	still waiting when the lab stopped it at 150 s
C# System.Net.Http.HttpClient	gave up after 100 s
Python requests	still waiting when the lab stopped it at 150 s

Only .NET has a default, and it’s 100 seconds. (The lab ran Go 1.26.2 and Python requests 2.31.0; the documentation quoted below is for Go 1.27.1 and requests 2.34.2, and says the same thing.) The others wait indefinitely for the answer, and their documentation says so plainly:

• Go: “A Timeout of zero means no timeout.” And http.DefaultClient, which http.Get uses, sets none. Go’s default transport does limit connecting to 30 seconds and a TLS handshake to 10, but not the wait for a reply, which is what hung here.
• Java: “The effect of not setting a timeout is the same as setting an infinite Duration, i.e. block forever.”
• Python requests: “By default, requests do not time out unless a timeout value is set explicitly.”
• gRPC: “By default, gRPC does not set a deadline which means it is possible for a client to end up waiting for a response effectively forever.”

You might hope the network would eventually notice. It won’t here, because nothing is wrong with the network. The connection is healthy; the server just isn’t answering. TCP only gives up on a connection when its own packets go unacknowledged, and even then Linux keeps retransmitting for a long time. Its manual says the default “corresponds to a duration of approximately between 13 to 30 minutes”. TCP keep-alive doesn’t help either. Linux doesn’t start probing until a connection has been idle for two hours, and even a client that probes sooner, as Go’s does, gets its probes answered: the silent server’s operating system is perfectly healthy.

Every call across a network needs a timeout you chose. Waiting forever holds a thread, a connection and a request’s memory for as long as the other side stays silent. When the other side is overloaded, that’s exactly when you can least afford it.

Choosing a timeout

“Set it to the p99” is common advice. None of the primary sources for this part says it. Here’s what they do say.

AWS starts from how many false timeouts you’re willing to accept:

A good practice for choosing a timeout for calls within an AWS Region is to start with the latency metrics of the downstream service. So at Amazon, when we make one service call another service, we choose an acceptable rate of false timeouts (such as 0.1%). Then, we look at the corresponding latency percentile on the downstream service (p99.9 in this example).

Notice the logic. A timeout at the p99 means about one call in a hundred that would have succeeded gets cut off, and very likely retried. The percentile is an output of how many of those you can afford, not a rule.

Google’s SRE book gives no formula, and admits it: picking a deadline “can be something of an art”. gRPC’s guidance is “an educated guess based on what you know about your system (network latency, server processing time, etc.), validated by some load testing.”

Three details that the percentile alone misses:

• What the timeout covers. AWS’s library tells of a 20-millisecond timeout that fired now and then, right after deployments, because the timer included setting up a new TLS connection, which the steady-state latency didn’t. Some implementations don’t cover DNS or TLS at all.
• A tight latency distribution. If p99.9 is close to p50, a small slowdown times out almost everything. AWS pads the value in that case.
• The whole request, not each hop. If the front end allows 30 seconds and spends 7 before calling the next service, that service has 23 left, not a fresh 30. Google calls this deadline propagation. Without it, deep services keep working on requests whose callers gave up long ago.

Retries multiply

A timeout turns “no answer” into “failed”. The natural next step is to try again. That’s often right: most failures are brief, and a second attempt succeeds. The trouble starts when every layer has the same idea.

The lab built a chain of real HTTP services, each calling the next, with the service at the bottom failing every request. It sent 200 requests in at the top and counted what arrived at the bottom.

Retrying layers	Every hop tries 3 times	Only the client retries	Every hop, 10% retry budget
1	3	3	1.11
2	9	3	1.23
3	27	3	1.365
4	81	3	1.515
5	243	3	1.68

If you guessed 243 for the second question, you did the arithmetic. It’s three to the power of the depth, and the lab confirms it: 48,600 requests reached the bottom for 200 at the top. AWS gives the same example:

Consider a system where the customer’s call causes a five-deep stack of service calls. It ends with a query to a database, and three retries at each layer. What happens when the database starts failing queries under load? If each layer retries independently, the load on the database will increase 243x, making it unlikely to ever recover.

Google’s SRE book has a version with 3 retries per layer, which is 4 attempts, three layers deep: 4 × 4 × 4 = 64. Same shape, different counting. Be clear in your own design reviews whether “3 retries” means three attempts or four.

The two other columns are the two standard ways out.

Retry in one place. If only the client retries and every service passes failure straight back up, the bottom sees 3 requests at any depth. AWS: “for low-cost control-plane and data-plane operations, our best practice is to retry at a single point in the stack.” Google goes further and says overloaded servers should reply with an “overloaded; don’t retry” error, so that layers above don’t try again.

A retry budget. Let every layer retry, but only while its retries stay under a fraction of its requests. Google’s SRE book uses 10%:

Secondly, we implement a per-client retry budget. Each client keeps track of the ratio of requests that correspond to retries. A request will only be retried as long as this ratio is below 10%.

In the lab that held five layers to 1.68 requests per request, where independent retries gave 243. A budget is a ratio over many requests, so it blocks retries until enough requests have gone by, and the lab sends a stream of 200 so it has something to be a ratio of. gRPC builds the same idea in as retry throttling, and AWS’s SDKs use a token bucket for it.

The budget in the lab is a few lines. Before any retry, a hop checks that one more retry would still keep retries under 10% of everything it has sent:

func (b *budget) allow() bool {
	b.mu.Lock()
	defer b.mu.Unlock()
	if float64(b.retries+1) > b.ratio*float64(b.requests+b.retries+1) {
		return false
	}
	b.retries++
	return true
}

Backoff, and why it needs jitter

Retrying at once is rude to a server that’s struggling. The standard courtesy is exponential backoff: wait a little, then twice as long, then twice that, up to a cap. It sounds like enough. The lab says otherwise.

It started 200 clients at the same moment against a real HTTP server that handles 10 requests at a time and turns the rest away with 429 Too Many Requests. Each client retries until it gets in. The five policies are the formulas from the simulator behind AWS’s article on the subject, with a 5 ms base and a 2 s cap.

Policy	Calls to the server	Until the last client got in	Busiest 10 ms, after the first 50 ms
No backoff	29,481	0.41 s	1,001 calls
Exponential	2,290	24.62 s	170 calls
Exponential, full jitter	1,358	0.91 s	55 calls
Exponential, equal jitter	1,329	1.09 s	79 calls
Decorrelated jitter	1,299	0.83 s	65 calls

Medians of five runs. Each run is 200 real clients on one machine, so the times vary: full jitter’s finish ranged from 0.85 s to 1.58 s across the five. The last column skips the first 50 ms. Every policy starts with the same burst, 200 clients at once, and what differs is what happens after it.

Exponential backoff without jitter took 24.62 seconds, more than twenty times longer than the slowest jittered policy. The reason is in the name of AWS’s article, and in its description of the same effect:

It’s obvious that the exponential backoff is working, in that the calls are happening less and less frequently. The problem also stands out: there are still clusters of calls. Instead of reducing the number of clients competing in every round, we’ve just introduced times when no client is competing.

All 200 clients were turned away at the same moment, so they all waited the same 5 ms, retried together, were mostly turned away together, waited 10 ms together, and so on. Each wave lets about 10 in. That’s about twenty waves, and once the delay reaches its 2-second cap, the waves are 2 seconds apart. Twenty waves of mostly waiting is where the 24 seconds went. The number is the product of this lab’s settings, but the mechanism isn’t. AWS’s Builders’ Library names it:

When failures are caused by overload or contention, backing off often doesn’t help as much as it seems like it should. This is because of correlation. If all the failed calls back off to the same time, they cause contention or overload again when they are retried. Our solution is jitter.

Jitter means randomising the wait. “Full jitter” picks a delay uniformly between zero and the exponential value. The clients drift out of step and the waves flatten: after the start, the busiest 10 ms went from 170 calls without jitter to 55 with full jitter.

No backoff at all finished first, in 0.41 seconds. But it made about 22 times as many calls as the jittered policies. Here, the server spent almost all its effort turning clients away. With a dependency that does real work per request, that’s the load that keeps it down.

The formulas, as the AWS simulator ships them, with v the capped exponential value:

• Full jitter: random.uniform(0, v).
• Equal jitter: v/2 + random.uniform(0, v/2), which keeps half the backoff fixed.
• Decorrelated jitter: min(cap, random.uniform(base, sleep * 3)), where each wait grows from the last one.

The libraries you’ll use differ in the details. gRPC’s retry specification changed in August 2024 from full jitter to “plus or minus 0.2” around the exponential value. .NET’s standard resilience handler uses exponential backoff with jitter on. Plain Polly defaults to a constant 2-second delay with jitter off, so if you build your own pipeline you have to switch it on.

When retries keep a system down

Everything so far has been about the cost of retrying while something is broken. The worst case is when retries keep it broken after the cause has gone.

Four researchers named this a metastable failure in 2021:

Metastable failures occur in open systems with an uncontrolled source of load where a trigger causes the system to enter a bad state that persists even when the trigger is removed. In this state the goodput (i.e., throughput of useful work) is unusably low, and there is a sustaining effect—often involving work amplification or decreased overall efficiency— that prevents the system from leaving the bad state.

The lab simulates it. This one is a simulation, not a timed run. The question is about how a queue behaves, and a simulated clock answers it exactly, without scheduler noise. The model is deliberately plain, and every assumption in it matters:

• a server with capacity for 1,000 requests a second, and one first-in, first-out queue with no limit on its length;
• new requests arriving at random at 700 a second, 70% of capacity;
• clients that give up after 1 second and, depending on the policy, send a retry at once, with no backoff;
• a server that doesn’t know a client gave up, so it serves abandoned requests anyway.

Then, at second 10, the server stops for 2 seconds, the way a long garbage-collection pause or a failover would stop it.

Policy	Back to normal
No retries	at second 14
3 retries	never, in 60 seconds
3 retries, 10% retry budget	at second 17
3 retries, server drops work it can’t finish in time	at second 12
3 retries, server drops only work that has already expired	never, in 60 seconds
3 retries, queue capped at 0.8 seconds of work	at second 13
3 retries, at 50% load instead of 70%	never, in 60 seconds
3 retries, at 30% load	at second 12

The table is one random arrival stream. Over five streams the same setups recovered within a second of these times, and the ones that never recovered never did in any stream.

Without retries, the backlog from the stall drains and the server is back two seconds later. With three retries, it never comes back. The stall is long over, and the server is running flat out the whole time, but none of its work is useful.

Here’s why. During the stall, requests wait more than a second, so their clients give up and retry. Now each request is in the queue twice, then three times, then four. The server works through a queue full of requests nobody is waiting for any more. By the time it reaches a live one, that one has waited over a second too, so its client has also given up and retried. Demand is now up to four times the original, well over capacity, and it stays there. 100,624 retries in 60 simulated seconds.

Headroom helps, but it isn’t a fix

At 50% load, the same stall still never recovers. At 30% it recovers at second 12. So spare capacity matters, and the 2022 study of metastable failures in real systems saw the same in its own experiments:

With more idle resources to handle the transient performance degradation, the system handled the trigger gracefully with only a temporary increase in latency.

But headroom only moves the line. The lab swept the length of the stall at four loads, over five random arrival streams each, all with 3 immediate retries:

Load	Recovers from every stall up to	Never recovers from a stall of
30%	2 s	3 s or more
50%	1.25 s	2 s or more
70%	1.1 s	1.25 s or more
90%	0.75 s	1.1 s or more

Between the two columns the outcome depended on the random arrivals: at 50% load, a 1.5 s stall recovered in one stream of five. At 70% load the line sits just past the client’s timeout, and it moves with the timeout. With timeouts of 1 and 2 seconds, a stall of 1.1 times the timeout recovered every time and one of 1.25 times never did; with a 0.5-second timeout, the 1.1 times stall recovered in four streams of five. More headroom lets the system ride out a longer stall. None of it made the system immune: every load tipped over eventually.

The same study found this sensitivity in its own experiments, “a 2%-decrease in available CPU or a 1-second increase in duration separated successful recovery from a metastable failure”, and identified retries as the sustaining effect in 11 of the 21 incidents in its table.

What gets you out

Three fixes recovered at 70% load, and so did more headroom:

• A retry budget. Retries are allowed while they stay under 10% of everything sent in the last 10 seconds. The simulation uses one budget shared by all the clients, as if they were one upstream service calling this one. That keeps demand under capacity, so the backlog drains. It recovered at second 17, three seconds later than no retries at all, because the budgeted retries still add work while the queue drains. It recovered 5 seconds after the stall ended, whether the stall came at second 10 or second 40.
• Dropping work that can’t finish in time. When a worker picks up a request, it checks whether the client will still be waiting when it’s done, and skips it if not. The server stops spending its capacity on the dead, and recovers the moment the stall ends, the fastest of the fixes at 70% load. The simulation makes this easy by giving the server each client’s exact deadline and a fixed 10 ms per request. A real server needs the deadline passed down to it and an estimate of how long the work will take.
• A bounded queue, if it’s short enough. Capping the queue at 0.8 seconds of work means an attempt that would wait too long is refused at once instead of served too late, and the refused client retries after 100 ms. It recovered at second 13. The cap has to be shorter than the client’s timeout. At 0.5 and 0.8 seconds of work it recovered with every wait the lab tried after a refusal: none, 100 ms and 1 second. At a full second of work, one timeout’s worth, a request at the back would wait the whole timeout before being served, and it recovered in every stream only when refused clients retried instantly, in one of five with a 100 ms wait, and never with 1 second.
• Enough headroom, for this stall. At 30% load it recovered at second 12.

The row that didn’t recover is worth a look. Dropping only work that has already expired never recovered. A first-in, first-out queue under overload always serves the oldest surviving request, which is the one closest to its deadline, and it finishes each of them just too late. The check has to be “can this finish in time”, not “has this already expired”. Google’s SRE book makes the general point: “you don’t get credit for late assignments with RPCs.”

And if a system is already stuck? The researchers who named the problem are blunt: “It will remain there until the load is significantly reduced or the retry policy is changed.” That means shedding load, flushing queues or changing client behaviour, not waiting for it to clear.

Questions this leaves

Which failures should a client retry? Ones where the server probably didn’t do the work, or where doing it twice is harmless. gRPC’s retry design puts it as “only status codes that indicate the service did not process the request should be retried”. A 429 or 503 that says “not now” is a good candidate, and .NET’s standard handler waits as long as a Retry-After header asks. A timeout on a request with side effects is only safe with an idempotency key.

How do a per-attempt timeout, a retry count and an overall deadline fit together? The overall deadline is what the caller can afford; the attempts have to fit inside it. .NET’s standard handler is a worked example: 10 seconds per attempt, 3 retries, and a 30-second total that ends the whole thing whatever the retries have left.

What if two requests with the same idempotency key arrive at once? The server has to serialise them, or it charges twice anyway. Stripe says that when a request “conflicts with another request that’s executing concurrently”, it doesn’t save a result for it “because no API endpoint initiates the execution”: the second one doesn’t run.

Does gRPC retry behind your back, like Go’s transport? Only in the narrowest case: “Only RPCs that failed due to low-level races are retried, and only if gRPC is certain the RPCs have not been processed by a server.” Anything more needs a retry policy you configure.

Two tools this part doesn’t cover in depth. A hedged request sends a second copy to another replica if the first is slow, and uses whichever answers first. Dean and Barroso report that waiting until the 95th-percentile latency before hedging “limits the additional load to approximately 5% while substantially shortening the latency tail”. A circuit breaker stops calling a dependency that keeps failing. AWS is cautious about them: “circuit breakers introduce modal behavior into systems that can be difficult to test”, and it limits retries locally with a token bucket instead. Part 33 comes back to circuit breakers.

The fallacies of distributed computing

Most of this part is one list, put together at Sun: eight assumptions that people new to distributed systems make, and that are false.

1. The network is reliable.
2. Latency is zero.
3. Bandwidth is infinite.
4. The network is secure.
5. Topology doesn’t change.
6. There is one administrator.
7. Transport cost is zero.
8. The network is homogeneous.

It’s usually credited to Peter Deutsch alone, and a page on Gosling’s blog at Sun, archived in 2009, lists all eight under Deutsch’s name. The fuller account is a 2004 article in which both men are quoted. It says the first four were Bill Joy’s and a colleague’s, collected by Gosling; Deutsch added the next three, in the early 1990s; and Gosling added the eighth “in 1997 or so”.

The first two are this part. “The network is reliable” is the payment server’s lost replies. “Latency is zero” is the client waiting forever for an answer that isn’t coming.

The .NET building blocks

.NET is the one platform in the lab with a default timeout, and it has the most built in:

• HttpClient.Timeout defaults to 100 seconds, as measured. Set it, or use a resilience pipeline that does.
• Microsoft.Extensions.Http.Resilience adds a standard handler: a 30-second total timeout, 3 retries with exponential backoff and jitter, a circuit breaker, and a 10-second timeout per attempt. It retries on 408, 429 and 5xx responses, and on connection errors and attempt timeouts. And it retries all HTTP methods, POST included, until you tell it not to.
• Polly, which that handler is built on, has different defaults if you use it directly: 3 retries, a constant 2-second delay and no jitter.

If you add the standard handler to a client that makes payments, either exclude POST from its retries or send an idempotency key the server honours.

Explain it like I’m ten

You post a letter to your gran asking if you can visit on Saturday. A week passes and no reply comes.

• Did she get it? Maybe the letter was lost. Maybe she replied and her letter was lost. Maybe she’s just slow. You can’t tell which from your side of the post box. That’s partial failure.
• So you send another letter. If she already said yes, she now has two letters, and if each one means “add a visit to the calendar”, she’s expecting you twice. Writing “this is about the same visit as my last letter” on the second one fixes that. That’s an idempotency key.
• How long do you wait before writing again? If you never decide, you wait forever. That’s a missing timeout.
• Now imagine a whole class writes to the same busy gran at once, and every child who hears nothing writes again exactly one day later. Her letterbox fills up in waves. If each child picks a random day to write again, the letters arrive at a steady pace. That’s jitter.
• And if the letterbox is so full that the replies take longer than a week, every child writes again, which makes the letterbox fuller, which makes the replies slower. The pile never goes down, even though gran is working as hard as she can. That’s a metastable failure.

The precise version

• The post box is the network, and “maybe lost, maybe slow” is the indistinguishability that Waldo and his co-authors, and FLP, both rest on.
• The note on the second letter is an idempotency key: the server stores the result per key and returns it for a repeat.
• The week you decide to wait is a timeout. The whole trip, if gran forwards your letter to your aunt with however much of the week is left, is a propagated deadline.
• Writing again one day later, then two, then four, is exponential backoff. Picking a random day within that is jitter.
• The pile that never shrinks is retry-sustained overload, and dropping letters too old to answer in time is deadline-aware load shedding.
• Where the analogy breaks: a real server can’t choose to read the newest letters first unless someone builds it to, and a real client can send its second request in microseconds, not days.

Trade-offs

• Retries buy availability with load. Every retry is a bet that the failure is brief. Measured: 243 times the load at a failing bottom service five layers down, when every layer made the bet.
• Retrying in one place is simple and puts every decision in one spot. Measured: 3 requests at the bottom at any depth. The cost is that a layer in the middle can’t retry a failure that only it knows is brief.
• A retry budget allows retries everywhere and caps their sum. Measured: 1.68 per request five layers deep. It blocks retries until enough requests have gone by, and it slowed recovery by three seconds in the simulation.
• A long timeout wastes resources; a short one makes false failures. And each false failure may be retried. AWS picks the percentile from how many false timeouts it can afford.
• No backoff finishes fastest and costs the most. Measured: 0.41 s and 29,481 calls, against 0.83 s and 1,299 calls with decorrelated jitter.
• Backoff without jitter is kind to the server and terrible for latency. Measured: 2,290 calls, and 24.62 s until the last client got in, because they stayed in step.
• Idempotency keys make retries safe, and cost storage and a lookup. Stripe keeps them for about 24 hours.

Common mistakes

• Calling across a network with no timeout. Measured: Go, Java and Python were still waiting at 150 seconds; .NET gave up at 100.
• Retrying a request that isn’t safe to repeat. Measured: 201 of 1,000 payments charged more than once.
• Assuming you know how many times your request was sent. Measured: 1,043 attempts in the program, 1,244 at the server. Go’s transport retried by itself.
• Letting every layer retry. Measured: 243 times the load at the bottom.
• Exponential backoff without jitter. Measured: 24.62 s against at most 1.09 s with jitter (medians of five runs), because everyone retried at the same moments.
• Setting timeouts to the p99 because someone said to. That’s a 1% false-timeout rate by construction, and each false timeout can become a retry.
• Serving work nobody is waiting for. Measured in the simulation: a server that only skipped requests already expired never recovered; one that skipped requests that couldn’t finish in time recovered immediately.
• Blaming the trigger. In a metastable failure the stall is over long before the outage is. The researchers who named it put it this way: “It is common for an outage that involves a metastable failure to be initially blamed on the trigger, but the true root cause is the sustaining effect.”
• Adding .NET’s standard resilience handler to a client that makes payments without excluding POST or adding idempotency keys.

Interview questions

Try to answer each one before opening the model answer.

1. What makes a distributed system fundamentally different from a single program?

Show a strong answer

• Partial failure. One part can fail while the rest keeps running, and there’s no central authority to say what failed.
• You can’t tell slow from dead, or a lost request from a lost reply. Waldo and colleagues put it as a network link failure being “indistinguishable from the failure of a processor on the other side of that link”.
• So every remote call has three outcomes: success, failure, and unknown. The design question is what you do with unknown.
• The consequences: timeouts to turn unknown into “probably failed”, retries to recover from brief failures, and idempotency so retries are safe.

Likely follow-up: “Can a better protocol remove the uncertainty?” No. The two generals argument shows no fixed-length exchange can guarantee agreement over a lossy channel, and FLP shows consensus can’t be guaranteed to finish if a crashed process can’t be told from a slow one. Real systems use timeouts, which are guesses.

2. A client times out waiting for a payment API. What should it do?

Show a strong answer

• Treat the outcome as unknown, not failed. The charge may have happened. Stripe tells its users to treat a 500 as indeterminate for the same reason.
• Retry with the same idempotency key and the same parameters, so the server can return the stored result instead of charging again. Measured: 201 of 1,000 payments charged more than once without keys, none with them.
• For a network error or a timeout, keep retrying with the same key within its lifetime, about 24 hours for Stripe, until you get a definite answer.
• A 500 is different: Stripe stores it under the key, so retrying the same key tends to return the same 500. Treat it as indeterminate and reconcile against the provider’s records rather than guessing.

Likely follow-up: “Why not generate a new key for the retry?” Because the first request may have succeeded, and a new key means the server will treat the retry as a new payment.

3. How do you choose a timeout?

Show a strong answer

• Start from the false timeouts you can accept, then read the matching percentile from the downstream service’s latency. AWS’s example: 0.1% false timeouts means p99.9.
• Pad it when the distribution is tight, where p99.9 is close to p50, or a small slowdown will time out almost everything.
• Check what the timer covers: connection setup, DNS and TLS may or may not be inside it.
• Propagate the deadline. Pass the remaining time down the call chain, so deep services stop working on requests nobody is waiting for.
• Never leave it unset. Measured: default clients in Go, Java and Python were still waiting at 150 seconds on a silent server.

Likely follow-up: “Why not just use the p99?” Then one call in a hundred that would have succeeded is cut off, and probably retried. The percentile should follow from the false-timeout rate you chose.

4. Five layers call each other in a chain, and each makes up to three attempts. What happens when the database fails?

Show a strong answer

• The load multiplies: three tries at each of five layers is 3^5 = 243 requests at the database per user request. Measured on a real chain: exactly 243.
• That load arrives exactly when the database is least able to take it, which can stop it from recovering at all.
• Fix one: retry at a single layer. Google’s advice is the layer immediately above the one that’s failing. Measured: 3 at any depth.
• Fix two: a retry budget, retrying only while retries are under about 10% of requests. Measured: 1.68 at five layers.
• Fix three: an “overloaded; don’t retry” error that tells upper layers not to try again.

Likely follow-up: “What about idempotency?” It’s a separate question. Idempotency makes a retry safe; it doesn’t make it cheap.

5. Why add jitter to exponential backoff?

Show a strong answer

• Because clients that fail together retry together. Backoff spaces the retries out in time, but not from each other, so they arrive in waves.
• Measured: 200 clients against a server that takes 10 at a time. Backoff without jitter took 24.62 s until the last client got in; full jitter took 0.91 s and decorrelated jitter 0.83 s.
• Full jitter picks a random delay between zero and the backoff value, which spreads the retries evenly.
• No backoff finishes fastest but makes about 22 times the calls. That’s affordable only if rejecting a request costs the server nothing.

Likely follow-up: “Is jitter enough on its own?” No. It reduces the retries but doesn’t cap them. You still need a limit on attempts and a budget.

6. What is a metastable failure?

Show a strong answer

• A system pushed into a bad state by a trigger, which stays bad after the trigger is gone, held there by a sustaining effect, most often retries.
• In our simulation, a server at 70% load stopped for 2 seconds. Without retries it recovered 2 seconds later. With 3 retries it never recovered.
• The mechanism: requests wait longer than the client timeout, clients retry, the queue fills with work nobody is waiting for, and new requests time out too.
• At 70% load the tipping point sat just past the client timeout: a stall of 1.1 seconds recovered; 1.25 seconds never did.
• Fixes that worked in the simulation: a windowed retry budget, dropping work that can’t finish in time, and a bounded queue. Dropping only work that had already expired did not work.

Likely follow-up: “Would more capacity have prevented it?” It helps, but only moves the line. At 30% load the same 2-second stall recovered; at 50% it didn’t. And at 30% a 3-second stall never recovered. Headroom buys you a longer stall, not immunity.

7. Which HTTP methods are safe to retry automatically?

Show a strong answer

• RFC 9110 defines GET, HEAD, OPTIONS and TRACE as safe, and PUT, DELETE and the safe methods as idempotent. Idempotent requests “can be repeated automatically if a communication failure occurs before the client is able to read the server’s response”.
• But the RFC doesn’t tell you to retry GET without limit. It says a client “SHOULD NOT automatically retry a failed automatic retry”, and a safe method’s implementation can still have side effects.
• POST is only safe to retry if the server makes it so, usually with an idempotency key.
• Know your libraries: Go’s transport retries idempotent requests, including POSTs with an idempotency key, by itself. .NET’s standard resilience handler retries all methods by default.

Likely follow-up: “Is an idempotent retry free?” No. It’s safe, but it’s still load, and every amplification problem in this part applies to it.

8. What are the fallacies of distributed computing, and which matter most day to day?

Show a strong answer

• The eight: the network is reliable; latency is zero; bandwidth is infinite; the network is secure; topology doesn’t change; there is one administrator; transport cost is zero; the network is homogeneous.
• Attribution: put together at Sun. Peter Deutsch wrote down the list of seven in the early 1990s; the first four are credited to Bill Joy and a colleague, and James Gosling added the eighth in about 1997.
• Day to day, the first two: an unreliable network forces timeouts and idempotent retries; non-zero latency forces deadlines, and an interface design that doesn’t make a remote call look like a local one.

Likely follow-up: “How does ‘topology doesn’t change’ bite?” Cached DNS answers, pinned connections to a server that’s been replaced, and client-side lists of hosts that go stale.

Sources

• Labs: system-design/checks/part23_retries/run.py. go/ measures:
• retry amplification through a chain of real HTTP services;
• five backoff policies with 200 clients against a server that serves 10 at a time;
• a discrete-event simulation of metastable failure, with a sweep of the stall length;
• lost responses with and without idempotency keys.

timeouts/ points the default HTTP clients of Go 1.26, JDK 25, .NET 10 and Python requests at a server that never answers. – Waldo, Wyant, Wollrath and Kendall, A Note on Distributed Computing, Sun Microsystems Laboratories TR-94-29, 1994. – Leslie Lamport, email to DEC SRC, 28 May 1987, and his publications page. – Jim Gray, Notes on Data Base Operating Systems, 1978, §5.8.3.3, the generals paradox. – Dean and Barroso, The Tail at Scale, CACM, 2013, for hedged requests. – Fischer, Lynch and Paterson, Impossibility of Distributed Consensus with One Faulty Process, JACM 32(2), 1985. – AWS: – Marc Brooker, Timeouts, retries and backoff with jitter, Amazon Builders’ Library, quoted from AWS’s own 2019 PDF. – Marc Brooker, Exponential Backoff And Jitter, 2015, with its simulator. – Google: – Handling Overload and Addressing Cascading Failures, Site Reliability Engineering. – gRFC A6: client retries, and gRPC’s guides on deadlines and retries. – Bronson, Aghayev, Charapko and Zhu, Metastable Failures in Distributed Systems, HotOS 2021; Huang et al., Metastable Failures in the Wild, OSDI 2022. – HTTP and TCP: – RFC 9110 §9.2 on safe and idempotent methods. – RFC 1122, RFC 6298 and RFC 9293 for TCP’s own timers. – Linux tcp(7). – Client defaults: – Go’s net/http client.go and transport.go. – Java’s HttpRequest.Builder. – Python requests’ advanced usage. – .NET’s HttpClient.Timeout, HTTP resilience and Polly retry. – Stripe’s idempotent requests and low-level error handling. – The fallacies: James Gosling’s page, and Ingrid Van Den Hoogen, Deutsch’s Fallacies, 10 Years After, Java Developer’s Journal, 2004.

What to remember

• A remote call can succeed, fail, or tell you nothing. Design for the third.
• A missing answer doesn’t mean the work didn’t happen. Measured: 201 of 1,000 payments charged more than once by honest retries.
• Idempotency keys make retries safe: none charged twice with them.
• Libraries retry too. Go’s transport sent 201 retries the program never made.
• Default clients wait: Go, Java and Python were still waiting at 150 seconds, and .NET gave up at 100.
• Choose a timeout from the false timeouts you can afford, and pass the remaining deadline down.
• Retries multiply through layers: 243 times at five. Retry in one place, or budget them.
• Backoff needs jitter, or clients retry in step: 24.62 s against at most 1.09 s (medians of five runs).
• Retries can keep a system down after the fault has gone. Where it tips depends on the client’s timeout and your headroom.

Every retry is a bet that the failure is brief. Make sure it’s safe to lose, and make sure you aren’t the only one placing it.