Replication: What Synchronous Actually Promises

Leader-follower replication, lag, stale reads and split brain, measured on a real PostgreSQL primary and replica — including a three-second partition that cost 2,342 acknowledged writes, and what the manual actually guarantees instead.

Replication is two problems wearing one word. The first is copying data to another machine. That part is solved, and every database ships it. The second is deciding what you promise a client while the copy is in flight — and that part is a choice you make, whether or not you know you are making it.

This part runs a real primary and a real replica in two containers, cuts the network between them, kills the primary, and promotes the replica. Every number below is what those two servers did.

Try this first

Your primary is cut off from its replica for three seconds. It keeps taking writes the whole time, acknowledging each one. Then it dies, and you promote the replica.

How many of those acknowledged writes are on the replica? Write down your answer for asynchronous replication and for synchronous.

What the manual actually promises

Start here, because almost everyone states this wrongly, including me before I read it properly.

“Synchronous replication means zero data loss” is not what PostgreSQL says. What it says is narrower and much more interesting:

There is no way to be certain that all standbys have received all outstanding WAL data at time of the crash of the primary. Some transactions may not show as committed on the standby, even though they show as committed on the primary. The guarantee we offer is that the application will not receive explicit acknowledgment of the successful commit of a transaction until the WAL data is known to be safely received by all the synchronous standbys.

Read that twice. The guarantee is about acknowledgement, not about the state of the database. A transaction can be committed on the primary and missing from every standby — what synchronous replication promises is that you were never told it succeeded.

That sounds weaker than the folklore version. It is exactly the promise you need, though, because it is the one your application can act on. You cannot build on “the data is definitely somewhere”. You can build on “if I was told yes, it survives”.

The parameter everybody forgets

Before any of the rest of this is usable: replication is asynchronous until you name a standby to wait for. Two settings are involved and only one of them is famous.

synchronous_commit defaults to on, which sounds synchronous and is not, on its own, anything to do with replication. What decides whether the primary waits for another server is synchronous_standby_names, which is empty by default. The manual is explicit about what that empty value does to the rest:

If synchronous_standby_names is empty, the only meaningful settings are on and off; remote_apply, remote_write and local all provide the same local synchronization level as on.

So a reader who sets synchronous_commit = remote_apply and changes nothing else has changed nothing at all. Every “synchronous” row below assumes a standby has been named.

The levels, and what each one waits for

With a standby named, synchronous_commit has five settings, and the differences between them are precise. Here is the manual’s own capability table, rebuilt cell by cell:

synchronous_commit	local durable commit	standby durable after a PostgreSQL crash	standby durable after an OS crash	standby query consistency
remote_apply	✓	✓	✓	✓
on (the default)	✓	✓	✓
remote_write	✓	✓
local	✓
off

Rebuilt from Table 19.1 in the PostgreSQL 18 manual.

Two things people get wrong, both visible in that grid.

off has no ticks at all — not even local durability. It is the only level that doesn’t wait for the primary’s own flush to disk. And the manual is careful about what that costs: “The risk that is taken by using asynchronous commit is of data loss, not data corruption… The net effect is therefore loss of the last few transactions.” You lose a suffix of the commit order, never a hole in the middle.

on, the default, does not give you standby query consistency. It waits for the standby to have flushed the commit record, not to have applied it. A transaction the primary has acknowledged may not yet be visible to a query on the standby. Only remote_apply claims that column, and the manual describes it with a hedge worth keeping: it “allows for load balancing with causal consistency” — “in simple cases”.

And remote_write is the one that gets flattened most. The manual: it “ensures data preservation if a standby instance of PostgreSQL crashes, but not if the standby suffers an operating-system-level crash because the data has not necessarily reached durable storage on the standby.”

What each level costs

Level	Waits for a standby	Best transaction time	Spread over 5 runs	Throughput
off	no	1.36 ms	0.25 ms	733 tps
local	no	1.39 ms	0.19 ms	720 tps
remote_write	yes	2.26 ms	0.22 ms	443 tps
on	yes	2.30 ms	0.59 ms	435 tps
remote_apply	yes	2.17 ms	0.53 ms	461 tps

pgbench -N, one client, five runs per level interleaved with the others, best run reported. A pgbench -N transaction is BEGIN, UPDATE, SELECT, INSERT, END — these are transaction times, not the cost of a commit alone. Look at the difference between the groups, not the absolute numbers.

The shape is what to take, not the digits. The two levels that do not wait sit at about 1.4 ms; the three that do sit at about 2.3 ms. Between the groups the difference is roughly 1.7 times the transaction time and about 1.7 times the throughput. Within each group the differences are smaller than the run-to-run spread, so this measurement cannot rank them.

That last point is worth dwelling on, because the manual says remote_apply “will cause much larger commit delays than previous settings since it waits for WAL replay” — and here it did not. Both servers are on one machine, so replay happens in microseconds and there is nothing for the extra wait to cost. Across a real network, where the round trip dominates, the manual’s warning is the one to believe and this table is not.

Treat all of these as a floor. No network, no distance, no other tenants, and pgbench running inside the primary’s own container.

One documented detail worth knowing: this is a per-transaction setting. “It is therefore possible, and useful, to have some transactions commit synchronously and others asynchronously.” The money moves synchronously; the analytics event doesn’t have to.

Replication lag, and what the columns mean

under a 6-second pgbench load, sampled 114 times
peak write lag       8.26 ms   (written on the standby)
peak flush lag       9.37 ms   (flushed to the standby's disk)
peak replay lag     10.48 ms   (visible to queries on the standby)
furthest behind     1,464 bytes of what the primary had already sent

pg_stat_replication reports three lags, and they map exactly onto the durability levels: write_lag to remote_write, flush_lag to on, replay_lag to remote_apply. So the columns are not three views of one number; they are the three different questions you might be asking.

The manual pre-empts the most common misreading of them, which is that an idle system should show zero:

In particular, when the standby has caught up completely, pg_stat_replication shows the time taken to write, flush and replay the most recent reported WAL location rather than zero as some users might expect

Under this load the replica stayed within about 10 milliseconds, and never more than 1,464 bytes behind what the primary had already sent. Note that last phrase: this measures replay against what was sent, which does not include WAL the primary had generated but not yet transmitted. And these are two servers on one machine, so it is the best case by a wide margin.

That is the normal picture, and it is precisely why the failure below surprises people: lag is tiny right up until it is not bounded at all.

Reading from the replica

If lag is a few milliseconds, reading from a replica is fine — right up until you read something you just wrote.

When	synchronous_commit	Stale reads	Gap from commit to read
idle	off	102 of 200 (51%)	1.13 ms
idle	on	0 of 200 (0%)	1.23 ms
idle	remote_apply	0 of 200 (0%)	1.23 ms
under load	off	115 of 200 (57.5%)	1.50 ms
under load	on	1 of 200 (0.5%)	1.51 ms
under load	remote_apply	0 of 200 (0%)	1.55 ms

The clock starts when the commit is acknowledged, so that last column is the gap a real client would experience between being told “done” and asking the replica. At the asynchronous setting, a read 1.1 to 1.6 milliseconds later missed the write about half the time, and more often under load.

Now look at the on row under load: one stale read in two hundred. That is not a bug, it is the guarantee working exactly as documented. on waits for the standby to have flushed the commit record, not to have applied it, so a read can arrive in the gap between those two things. The capability table above says as much — on has no tick in the standby-query-consistency column. Our idle runs showed 0%, which is what “permitted but rare” looks like until it isn’t.

remote_apply is the level that promises otherwise, and it showed no stale read in 400 attempts here.

One caveat that applies to this whole table: both servers are on one machine. Flush and apply are microseconds apart on loopback, which is why on got away with it 399 times out of 400. Across a real network that gap is wider and so is the window.

The other thing to notice is how easily this hides. Our probe holds both connections open, so the gap is a millisecond or two. Open a fresh connection for each read — as a naive test harness does — and you add far more delay than the replication lag you were trying to catch, the replica is always caught up by the time you look, and the anomaly is invisible. That is how it reaches production.

This pattern has a name, from the session-guarantees literature rather than from any database manual: read-your-writes. Terry and colleagues’ definition is narrower than people assume — it is about writes being reflected in what you subsequently read, and explicitly not a promise about reading back a particular value: “Applications are not guaranteed that a Read following a Write to the same data item will return the previously written value.” PostgreSQL’s documentation does not use the phrase at all.

The failure that costs you data

Start with the simple version: kill a healthy primary and promote the replica.

no partition: a healthy primary, killed, and the replica promoted
acknowledged by the primary   400
found on the promoted replica 400
lost                          0

Nothing lost — in this run. Do not generalise that into a rule, because it isn’t one. The manual is clear that asynchronous replication loses “some transactions that were committed”, and that “The amount of data loss is proportional to the replication delay at the time of failover.” On a healthy pair on one machine that delay is milliseconds, so a kill catches almost nothing in flight. On a busy pair across a network it would.

What makes the loss large and certain is a primary that goes on acknowledging writes while the replica cannot hear it. So the real test cuts the network between them, keeps writing for three seconds, and only then kills the primary.

	synchronous_commit = off	synchronous_commit = on
acknowledged before the partition	500	500
acknowledged during the 3-second partition	2,342	0
write attempts that blocked	0	4
rows the primary held when it died	2,842	500
found on the promoted replica	500	500
acknowledged writes lost	2,342	0

At the asynchronous setting the primary acknowledged 2,342 writes during those 3 seconds, and every one of them died with it. That number is three seconds multiplied by whatever this laptop could write — not a property of asynchronous replication. What is the property is the shape: everything acknowledged during the partition, however long it lasts and however fast you write.

Nothing was corrupted. The promoted replica is perfectly consistent; it is simply missing two thousand transactions that a client was told had succeeded.

At the synchronous setting the primary acknowledged nothing during the partition. The first write blocked, and so did three more attempts on fresh connections — the fourth client got exactly the same treatment as the first. And look at the row above the total: the primary held 500 rows when it died, exactly what it had acknowledged. It was not sitting on committed-but-unacknowledged work; it was simply stopped.

The manual is blunt that this is the intended behaviour and not an edge case: “Such transaction commits may never be completed if any one of the synchronous standbys should crash.” There is no documented timeout on that wait.

Which is worth contrasting with the most common alternative. MySQL’s semisynchronous replication does have a timeout, and degrades: “If a timeout occurs without any replica having acknowledged the transaction, the source reverts to asynchronous replication.” Two reasonable designs with opposite failure modes. One stops serving; the other stops promising.

Choosing between them is choosing which failure you would rather explain.

Split brain, and what stops it

Our lab takes the asynchronous failover above — the one that lost 2,342 writes — and simply starts the dead primary again.

after the asynchronous failover above, the old primary was started again
old primary in recovery?  no
new primary in recovery?  no
both accepted a write?    yes
rows on the old primary   2,843
rows on the new primary   501

Two servers, both accepting writes, diverging: 2,843 rows on one and 501 on the other. Neither knows the other exists.

PostgreSQL’s documentation is honest about this: it “does not provide the system software required to identify a failure on the primary and notify the standby database server.” The word “fencing” doesn’t appear in a clustering sense anywhere in the manual, and a search of the version 18 documentation for “split brain” returns no hits at all. What the high-availability chapter does discuss is STONITH — shoot the other node in the head — as something you arrange around the database, not something it does for you.

Which is the real lesson of this section: the thing that prevents split brain is never the database. It is whatever fences the old primary — the thing that guarantees it is powered off, or has lost its floating IP, or cannot reach the disk — before anything else is promoted.

And note that a quorum doesn’t make this go away either. MongoDB, which does ship the machinery, documents that “two members in a replica set transiently believe that they are the primary”; a majority write concern stops the loser committing and rolls back what it had accepted. That is a much better outcome than our two diverging servers, but “the quorum prevented it” is not what happened.

The shapes: leader-follower, multi-leader, leaderless

Everything above is leader-follower: one node takes writes, others copy. It is the default for a reason — there is exactly one place where write order is decided, so there is nothing to reconcile.

Multi-leader puts writers in more than one place, usually for latency or for regional independence. The copying is the easy part; the hard part is what happens when two leaders accept conflicting writes. Somebody has to decide, and each option costs something: last-write-wins discards data according to clocks that disagree, application-level merges push the decision to you on every write path, and conflict-free replicated data types remove the decision entirely for the data shapes they model — which is a real answer when your data is a set or a counter, and no answer at all when it is an invoice.

Leaderless replication, in the Dynamo lineage, has clients write to several nodes and read from several, with N replicas, W acknowledgements to write and R to read. The familiar rule is that R + W > N gives you strong consistency. What is striking is that the Dynamo paper doesn’t claim that. It says the protocol is “similar to” quorum systems and “yields a quorum-like system”, and then:

[it] does not enforce strict quorum membership and instead it uses a “sloppy quorum”

A sloppy quorum, in the paper’s own words, means “all read and write operations are performed on the first N healthy nodes from the preference list, which may not always be the first N nodes encountered while walking the consistent hashing ring” — so the set of nodes that acknowledged your write and the set you read from need not overlap at all.

The paper is equally candid about what its conflict resolution costs, which is a separate mechanism and a separate bill. Because divergent versions are reconciled by the client rather than by the store, “an “add to cart” operation is never lost. However, deleted items can resurface.” That is the price of merging, not of the quorum — worth keeping the two straight, because they are usually quoted as one.

That’s not an argument against leaderless replication. It’s an argument for reading what a system promises rather than what its reputation promises.

Explain it like I’m ten

You and a friend both keep a copy of the class register.

• Leader-follower: only you write in it, and you read your changes out to your friend. If you’re away, your friend can still tell people what’s in it — just possibly missing the last line you added.
• Asynchronous: you read the change out after you’ve told the teacher it’s done. If you vanish mid-sentence, the teacher thinks it’s recorded and your friend never heard it.
• Synchronous: you wait for your friend to say “got it” before telling the teacher. Safer — and if your friend goes home, you can’t tell the teacher anything at all.
• Split brain: you’re both away for a bit, you each think you’re in charge, and you each write different things in your own copy. Now there are two registers and no way to say which is right.

The last one is why someone has to be able to say, definitely, that the other person has stopped writing.

The precise version

• What the primary tells the client is an acknowledgement, and the guarantee is about that, not about where the bytes are.
• The distance between the two copies is replication lag, and it is fine until a partition makes it unbounded.
• Reading your own write from a lagging replica is a broken read-your-writes guarantee.
• Two nodes both accepting writes is split brain, and the cure is fencing, which lives outside the database.
• Where the analogy breaks: your friend can shout “stop!”. Two servers can’t tell a dead peer from a slow one — that’s the whole problem.

Trade-offs

• Asynchronous buys latency and sells durability, and the bill is everything acknowledged during the outage: 2,342 writes in our three-second partition, on a laptop.
• Synchronous buys durability and sells availability. Our primary stopped accepting writes entirely while the standby was unreachable, with no timeout.
• The guarantee is about acknowledgement. Neither setting promises the data is on two machines; one promises you were never told it was.
• Per-transaction durability is the underused answer. Payments synchronous, telemetry asynchronous, one database.
• Replica reads trade freshness for capacity, and the staleness hides in testing, because anything that adds delay between the write and the read makes it disappear.
• Quorums change the shape of the failure, not its existence. Read what the system documents, not what its reputation says.
• Fencing is not optional, and it is not the database’s job. Without it, promotion is how you get two primaries.

Common mistakes

• “Synchronous means zero data loss.” It means you are not told yes until the standby has it. The manual says outright that transactions may show as committed on the primary and not on the standby.
• Turning on synchronous replication with one standby. Now the standby’s availability is the primary’s availability, and there is no timeout.
• Reading your own write from a replica. Measured: stale most of the time at the asynchronous setting.
• Testing replica reads with a fresh connection each time. Connection setup adds far more delay than the lag you are hunting, so the replica has always caught up by the time you look.
• Assuming on makes replica reads consistent. It doesn’t; only remote_apply claims that, and the manual hedges even then.
• Bringing an old primary back without fencing it. Our lab killed the primary, promoted the replica, then simply started the old one again — and got two servers both taking writes.
• Believing a quorum prevents two primaries. It prevents the loser’s writes from committing, which is not the same thing.
• Treating lag as a property of the system. It is a property of the moment. It is milliseconds until it is unbounded.

Interview questions

Try to answer each one before opening the model answer.

1. What does synchronous replication actually guarantee?

Show a strong answer

• That you are not told a commit succeeded until a standby has the WAL. It is a promise about acknowledgement, not about the state of the cluster.
• PostgreSQL says so explicitly: transactions may show as committed on the primary and not on the standby after a crash; the guarantee is about what the application was told.
• That is the useful form, because application logic can rely on “if I got a yes, it survives”.
• The cost is availability: with no synchronous standby available the primary waits, and PostgreSQL documents that such commits “may never be completed”. There is no timeout.
• And the levels differ: remote_write survives a PostgreSQL crash on the standby but not an OS crash; on survives both; only remote_apply makes it visible to standby queries.

Likely follow-up: “So how do you get closer to zero data loss?” More than one synchronous standby, so losing one doesn’t stop the primary, plus fencing so a stale primary cannot come back. Note “closer”: the guarantee stays one about acknowledgement, and the manual says outright there is no way to be certain every standby has everything at the moment the primary dies.

2. Your replica is 20 seconds behind. What do you do?

Show a strong answer

• Find out which lag, because write, flush and replay lag mean different things — receipt, durability and visibility.
• Check whether it is falling further behind or holding steady. Growing lag is an emergency; steady lag is a latency or a replay problem, and the causes below tell you which.
• The usual causes: a long-running query on the standby blocking replay, a write burst, a slow disk on the replica, or the network.
• Decide what depends on it: if replicas serve reads, you are serving 20-second-old data to users right now.
• And if it’s a synchronous standby, the primary is paying that latency on every commit — or has stopped.

Likely follow-up: “Why would a query on the standby block replay?” Replay may need to remove rows a standby query is still reading; the standby either delays replay or cancels the query.

3. How do you stop a user reading stale data after their own write?

Show a strong answer

• Route that user’s reads to the primary for a window after they write. Simple, effective, and the usual answer.
• Or use remote_apply, which waits until the standby has applied the commit — but pays for it on every commit, not just the ones that matter.
• Or track the write position and have the read wait until the replica has reached it. More work, better targeted.
• Measured: at the asynchronous setting a read 1.1 ms after the commit missed it about half the time. It is not a rare race.
• And test it properly: anything that delays the read — a fresh connection, a sleep, a slow assertion — hides the problem completely.

Likely follow-up: “What about monotonic reads?” Pin a session to one replica, or you can see time go backwards by hitting two replicas at different positions.

4. Walk me through a failover.

Show a strong answer

• Detect that the primary is gone, which is the hard part — a partition looks exactly like a death.
• Fence it: make sure the old primary cannot accept another write. Power, network, or storage. This step is the one people skip.
• Promote a standby, choosing the most advanced one if you have several.
• Redirect clients, usually by moving an address rather than by changing application config.
• Then decide what to do with the old node: rejoining it needs its divergent WAL dealt with, which is what pg_rewind is for.

Likely follow-up: “What did you lose?” Everything acknowledged but not replicated — in our three-second partition, 2,342 writes. With synchronous replication nothing was acknowledged during the partition at all, so there was nothing to lose.

5. What is split brain, and what prevents it?

Show a strong answer

• Two nodes both believing they are the primary, both accepting writes, diverging.
• We reproduced it in one command: restart the old primary after promoting the replica and both accept writes.
• What prevents it is fencing, not the database. PostgreSQL documents that it “does not provide the system software required to identify a failure on the primary and notify the standby database server”.
• A quorum helps but doesn’t eliminate it: MongoDB documents that two members can transiently both believe they are primary; majority write concern stops the loser committing and rolls back what it accepted.
• So the operational question is always: what makes it impossible for the old primary to serve one more write?

Likely follow-up: “How would you fence in a cloud?” Detach the volume, revoke the security group, or release the floating IP — something the old node cannot undo by itself.

6. When would you choose multi-leader replication?

Show a strong answer

• When writers must be near users in more than one region, or when regions must keep working independently during a partition.
• The copying is easy; the conflicts are the product. You are signing up to decide what happens when two leaders accept incompatible writes.
• The options are all trades: last-write-wins discards data by clock skew, application merges put the burden on you, CRDTs solve it for the data types they cover.
• Offline-capable clients are the honest case — a phone that writes while disconnected is a leader whether you planned it or not.
• Do not choose it for write throughput without pricing the reconciliation.

Likely follow-up: “Why is last-write-wins dangerous?” It depends on clocks agreeing, and Part 25’s whole subject is that they don’t.

7. Does R + W > N give you strong consistency?

Show a strong answer

• Only if the read set and the write set really intersect, which requires a strict quorum over a fixed set of replicas.
• Dynamo doesn’t claim it. The paper says the protocol is “similar to” quorum systems and “yields a quorum-like system”, then that it “does not enforce strict quorum membership and instead it uses a “sloppy quorum””.
• A sloppy quorum uses the first N healthy nodes, which “may not always be the first N nodes encountered while walking the consistent hashing ring” — so the sets may not overlap.
• And the paper is candid about its reconciliation, separately: because clients merge divergent versions, “deleted items can resurface”. That is the cost of merging, not of the sloppy quorum — keep the two apart.
• The lesson is general: read the system’s own statement of its guarantee, not the rule of thumb that grew up around it.

Likely follow-up: “What’s hinted handoff?” A node that isn’t a designated replica accepts the write and passes it on later — the mechanism that makes the quorum sloppy.

8. How would you decide the replication setup for a new service?

Show a strong answer

• Start from what you must never lose, and set that class of transaction synchronous. Not the whole database — the transactions.
• Then ask what your availability requirement is, because synchronous with one standby makes the standby a single point of failure.
• Two synchronous standbys, or quorum commit, if you need both durability and the freedom to lose a replica.
• Decide what reads from replicas, and accept that those reads are stale by an unbounded amount during an incident.
• Then build the fencing and rehearse the failover, because the parts that go wrong are the operational ones, not the settings.

Likely follow-up: “How would you test it?” Exactly as this part’s lab does: partition the replica, keep writing, kill the primary, and count what survived.

Sources

• Lab: system-design/checks/part20_replication/postgres.py — two PostgreSQL 18 containers on a private network with streaming replication, measuring commit latency at every synchronous_commit level with pgbench, replication lag under load, read-your-writes staleness with held-open sessions, and a failover in which the replica is partitioned away before the primary is killed
• PostgreSQL 18: WAL configuration (synchronous_commit and Table 19.1), asynchronous commit, log-shipping standby servers, hot standby, the statistics views, high availability
• G. DeCandia et al., Dynamo: Amazon’s Highly Available Key-value Store, SOSP 2007; D. Terry, A. Demers, K. Petersen, M. Spreitzer, M. Theimer and B. Welch, “Session Guarantees for Weakly Consistent Replicated Data”, Xerox PARC (the copy consulted carries no venue or date); D. Gifford, “Weighted Voting for Replicated Data”
• MySQL 8.4: semisynchronous replication (via the Internet Archive; the live page would not serve); MongoDB: write concern, replica set elections

What to remember

• The guarantee is about acknowledgement. Synchronous replication promises you were not told yes prematurely, not that the data is everywhere.
• off doesn’t even wait for a local flush. on doesn’t make replica reads consistent. Only remote_apply does, and the manual hedges it.
• A synchronous primary with no standby waits, and PostgreSQL documents that such commits “may never be completed”. There is no timeout.
• Replication lag is milliseconds until a partition makes it unbounded.
• Reading your own write from a replica is stale about half the time at the asynchronous setting, and even on allowed one in two hundred. A slow test hides all of it.
• Split brain is prevented by fencing, which is not the database’s job. Promotion without it is how you get two primaries.
• Durability is a per-transaction setting. Use it that way.

Replication doesn’t decide whether you lose data. It decides whether you promised not to.