software architecture

We killed Kafka consumers 310 times at random moments and counted every record lost or processed twice. The order of two lines decides which you get. Queues vs logs, delivery semantics, Kafka’s exactly-once, ordering, poison messages and backpressure.

We killed, froze and cut off the leader of a real etcd cluster, 66 times. A killed one was replaced in a median 1.2 seconds, and two nodes once both called themselves leader for about 0.8 s. How Raft elects, replicates and commits, and why it stays correct.

This laptop’s clock, kept in sync by NTP, was tens of milliseconds off, and 187 ms off after a sleep. We measured what a gap like that does to a database that keeps the last write, then built the clocks that fix it: Lamport, vector, hybrid, TrueTime.

One etcd node cut off from its cluster, measured. During the cut its linearizable reads and writes got no answer, and its serializable reads all answered out of date. Plus what CAP proves, what “2 of 3” gets wrong, and what consistency costs day to day.

A remote call can succeed, fail, or tell you nothing. Measured on real HTTP: lost replies charge customers more than once, three of four default clients wait 150 s or more, and retries multiply 243 times over five layers, then keep a simulated server down.

Cache-aside to write-behind, what Redis 8 really does when it fills up, the invalidation race and the lease that closes it, three fixes for a stampede measured against doing nothing, and the HTTP caching headers most people misread.

Range against hash, consistent hashing, virtual nodes and hot keys — measured by placing a million keys four ways and counting what moves, plus what partitioning costs a query inside one PostgreSQL 18 database.

Leader-follower replication, lag, stale reads and split brain, measured on a real PostgreSQL primary and replica — including a three-second partition that cost 2,342 acknowledged writes, and what the manual actually guarantees instead.

ACID, the anomalies and MVCC, with six classic anomalies replayed at three isolation levels by two live PostgreSQL 18 sessions — including the one that leaves nobody on call at the level almost everyone runs.

Composite column order, covering indexes, why an index gets ignored and how to read EXPLAIN — measured on one PostgreSQL 18 table in pages touched, including three rules of thumb the measurements contradict.