#Replication and consistency

Replication gives you three different things — durability, read scale, and availability — and the topology you choose depends on which one you actually need. Saying which is a good way to open.

#1 · Why replicate

These pull in different directions. Read scale wants many async replicas; durability wants synchronous acks; low-latency writes want the leader nearby, which cannot be true for everyone at once.

#2 · Leader-follower (primary-replica)

The default, and the right first answer.

        writes
          |
          v
      [ LEADER ] --replicate--> [ follower ] <- reads
                 --replicate--> [ follower ] <- reads

#Synchronous, asynchronous, or semi-synchronous

Fully synchronous replication to all replicas is a trap and worth naming as one: it makes your write availability the product of every node's availability, so adding replicas makes writes less available. Semi-sync — ack once at least one replica has it — gives you durability against single-node loss without that coupling.

#3 · Replication lag

The practical consequence of async replication, and the source of the bugs users actually notice.

#Read-your-writes

A user posts a comment, the write goes to the leader, the page reloads and reads from a replica that has not caught up — the comment they just wrote is gone.

"Read from the leader for a short window after a write" is the pragmatic answer, and scoping it to the user's own mutable data keeps leader load small.

#Monotonic reads

Two successive reads hit different replicas with different lag, and time appears to go backwards — a comment appears, then vanishes. Fix: route a given user consistently to one replica.

#Consistent prefix

Causally related writes are seen out of order — the reply arrives before the question. Fix: keep causally related writes in the same partition, or track causality explicitly.

Naming these three by name — read-your-writes, monotonic reads, consistent prefix — is a strong signal, because they are the three anomalies that make "eventually consistent" concrete rather than hand-wavy.

Monitor replication lag and alert on it. It is the single most useful health metric for a replicated database, and it is the thing that silently degrades before it visibly breaks.

#4 · Multi-leader

Writes accepted at several leaders, replicated to each other.

Use for: multi-region write latency, offline-capable clients, collaborative editing.

The cost is conflicts, which are unavoidable — two regions modify the same record concurrently and both are already committed.

Last-write-wins is the default in several systems and it loses data. Being able to say that — and that "last" is decided by clocks that are not synchronised — is exactly the kind of thing that separates candidates.

#5 · Leaderless and quorums

Dynamo-style: the client writes to several nodes and reads from several.

N = replicas,  W = write acks required,  R = read replies required

W + R > N  ->  read and write sets OVERLAP, so a read sees the
               latest write ("strong" quorum consistency)

N=3, W=2, R=2   the standard: survives one node down, either way
N=3, W=3, R=1   fast reads, writes fail if any node is down
N=3, W=1, R=1   fastest, weakest -- may read stale

W + R > N is the formula to know, and the intuition behind it is the pigeonhole principle: if the write touched W nodes and the read touched R nodes and W + R exceeds N, the two sets must share at least one node — and that node has the latest value.

The honest caveat, worth volunteering: quorums do not give you real linearizability. Concurrent writes, failed writes that partially succeeded, and recovery from backup all break the guarantee at the edges. It is stronger consistency, not strong consistency.

Supporting mechanisms: read repair (the reader notices a stale replica and fixes it), hinted handoff (another node holds a write for a down node), anti-entropy (background Merkle-tree comparison).

#6 · Failover

"The leader dies — what happens?" A very common probe, and the good answer admits the problems.

1. DETECT     heartbeat timeout (typically 10-30s)
2. ELECT      choose the most up-to-date follower
3. PROMOTE    reconfigure it as leader
4. REDIRECT   point clients and replicas at it

What goes wrong, and you should say it:

Fencing tokens are the split-brain answer. Each leadership term gets a monotonically increasing number, and storage rejects writes carrying an old token. A deposed leader that comes back cannot corrupt anything, because its token is stale. This is how you make "the old leader might still be alive" safe rather than merely unlikely.

#7 · Choosing, in the round

"Leader-follower with async replicas. Reads go to replicas, except a user's own data for thirty seconds after they write it — that goes to the leader, so they always see their own changes. Semi-sync so we don't lose acked writes to a single node failure. I'd alert on replication lag, and on failover I'd expect roughly thirty seconds of write unavailability plus a cold cache on the new leader."

That is a complete answer: topology, read routing, the read-your-writes fix, durability, monitoring, and the honest cost of failover.

#8 · Interview questions

#Stop condition

You know this block when you can:

1. explain why fully synchronous replication hurts availability,
2. name the three lag anomalies and fix read-your-writes,
3. derive W + R > N and state its honest limit,
4. walk the four failover steps and the four things that go wrong, and
5. explain fencing tokens.