#Caching

Caching appears in essentially every design round, which means saying "and we'll add a cache" is worth nothing. The score is in where, which pattern, how it is invalidated, and what happens when it fails.

#1 · Where caches live

There are six layers, and a good answer names the one it means.

flowchart LR
    B["Browser cache<br/>0ms"] --> C["CDN edge<br/>~20ms"]
    C --> G["API gateway<br/>cache"]
    G --> A["App-local<br/>in-process, ~0.1ms"]
    A --> D["Distributed cache<br/>Redis, ~1ms"]
    D --> E[("Database<br/>buffer pool")]

In-process caching is faster and much more dangerous. Every instance holds its own copy, so an invalidation must reach all of them and a user can see a value flip back and forth depending on which server they hit. It is the right answer for small, slow-changing config; the wrong answer for anything a user just edited. Saying that distinction out loud is a depth signal.

#2 · The read patterns

#Cache-aside (lazy loading) — the default

The application owns the logic.

def get_user(user_id):
    key = f"user:{user_id}"
    cached = cache.get(key)
    if cached is not None:
        return cached                      # hit

    user = db.query("SELECT ... WHERE id = ?", user_id)   # miss
    cache.set(key, user, ttl=300)
    return user

This is the right default and the right first answer. Say "cache-aside" by name.

#Read-through

The cache sits inline and fetches on miss itself. Simpler application code, but the cache becomes a hard dependency — if it is down, you are down.

#3 · The write patterns

This is where the interesting trade-off lives.

Delete the key; do not update it. Updating on write invites a lost-update race: two concurrent writers can leave the cache holding the older value permanently. Deleting means the next read repopulates from the source of truth. The worst case is an extra miss, not permanent corruption.

Write-back is the one to mention carefully. It is the fastest and the only one that can silently lose acknowledged writes when a cache node dies. Legitimate for view counters and metrics; unacceptable for anything a user would notice missing. Saying "write-back, but only because losing a few view counts is acceptable" is exactly the kind of scoped trade-off the round rewards.

#4 · Eviction and TTL

LRU's weakness is worth knowing: one large scan — a batch job reading every row — evicts the entire hot set. LFU or a segmented LRU resists this. Redis's allkeys-lru is approximate (it samples rather than maintaining an exact list), which is a good example of trading precision for O(1).

On TTLs:

• Always set one, even when you invalidate explicitly. It is the backstop for the invalidation you will eventually miss.
• Jitter it. A thousand keys written together with ttl=300 all expire in the same second. ttl = 300 + random(0, 60) spreads the reload.

#5 · The three failure modes

This section is the depth. Candidates who can name and fix these are visibly different from candidates who cannot.

#Stampede (dogpile)

A hot key expires; 10,000 concurrent requests all miss and all hit the database at once.

t=0    key "top_posts" expires
t=0.1  10,000 requests miss simultaneously
t=0.1  10,000 identical queries hit the database
t=0.2  database saturates -> everything is slow -> more requests pile up

Single-flight is the answer to give, and "serve stale while revalidating" is the one that impresses, because it keeps p99 flat during the refresh.

#Penetration

Requests for keys that do not exist — often an attack. Every one misses the cache and hits the database, which is exactly what the cache was meant to prevent.

A Bloom filter is the textbook answer here and is worth being able to justify: it can produce false positives (a wasted lookup) but never false negatives, so it can only ever say "definitely not present" — which is exactly the guarantee you need, in a fraction of the memory of the full key set.

#Avalanche

A large fraction of the cache disappears at once — a node dies, or many keys share an expiry — and the full read load lands on the database.

The question behind all three is the same: "what is your cache hit rate, and is the database sized for what happens when it drops?" A design that only works at 99% hit rate and collapses at 70% is fragile, and saying so about your own design is strong.

#6 · Invalidation

The hard part, and the honest answer is that there are only three strategies:

Versioned keys are underrated in interviews. Instead of invalidating, change the key: bump a version counter on write and every reader naturally computes a new key. There is no invalidation to get wrong, and no thundering delete. The cost is that the old entries linger until evicted.

For a group of keys, keep the version on the group:

key = f"feed:{user_id}:v{get_version(user_id)}"
# one INCR on the version invalidates every cached page of that feed

#7 · Hit rate

hit rate = hits / (hits + misses)

Effective latency is what actually matters:

90% hit:  0.9 x 1ms + 0.1 x 50ms  =  5.9 ms
95% hit:  0.95 x 1ms + 0.05 x 50ms =  3.5 ms
99% hit:  0.99 x 1ms + 0.01 x 50ms =  1.5 ms

Going from 90% to 99% roughly quarters your latency — and cuts database load by 10×. That second effect is the bigger one, and it is why hit rate is a capacity number rather than a performance nicety. Quoting this computation is a compact way to justify the whole cache tier.

If the hit rate is genuinely low (< 50%), the cache may be the wrong tool: the access pattern has no locality, and you are paying a network hop to miss.

#8 · What to say in the round

"Cache-aside in Redis for user and post lookups, TTL 5 minutes with jitter, and I delete the key on write rather than updating it — updating races. For the hot keys I'd use single-flight so an expiry doesn't stampede the database, and I'd rather serve stale for a moment than queue behind a refresh. Sizing assumes a 95% hit rate; if a cache node is lost, consistent hashing means we lose 1/N, and the database needs headroom for that."

Six decisions, each justified, in one breath — pattern, TTL, jitter, invalidation, stampede protection, and failure sizing. That is a complete caching answer.

#9 · Interview questions

#Stop condition

You know this block when you can:

1. name the six cache layers and the danger of the in-process one,
2. explain why you delete rather than update on write,
3. name all three failure modes and a fix for each,
4. justify a Bloom filter by its one-sided error, and
5. compute effective latency at 90% versus 99% hit rate.