#Queues and streams

Adding a queue is how you turn a synchronous failure into an asynchronous delay. That sentence is the whole justification, and it is also the whole cost — because the delay is now something you have to reason about.

#1 · What a queue actually buys

Four things, and a good answer names which one it wants:

And the costs, which you should volunteer:

"I'll make this async" is not free, and saying what it costs is the difference between using a queue and knowing why. The most common real consequence: the user posts something and does not see it, because their own write has not been fanned out yet.

#2 · Queue versus log

The distinction that matters most, and the one candidates blur.

TASK QUEUE (SQS, RabbitMQ)
  producer -> [ job job job ] -> consumer   consumes and DELETES
  - message is removed once acked
  - work is distributed; each job done once
  - no history, no replay
  - competing consumers scale trivially

LOG / STREAM (Kafka, Redis Streams, Kinesis)
  producer -> [ 0 1 2 3 4 5 6 7 ] partition, append-only, RETAINED
                      ^        ^
                  group A   group B     each tracks its OWN offset
  - messages persist for a retention window
  - many independent consumer groups read the same data
  - replay by resetting an offset
  - ordering guaranteed WITHIN a partition

The clearest way to choose: if a second team might later want the same events for a different purpose, you want a log. If the message is a unit of work that one worker should do once and forget, you want a queue. Replay alone often decides it — being able to reprocess a week of events after fixing a bug is worth a great deal.

#3 · Delivery guarantees

Exactly-once delivery is impossible; exactly-once processing is achievable. This is worth being able to say precisely, because it is a favourite probe.

The consumer must do two things:
  1. process the message
  2. acknowledge it

There is no way to make these atomic across two systems.

  ack first, then process   -> crash between = message LOST
  process first, then ack   -> crash between = message REDELIVERED

So you choose at-least-once (process, then ack) and make processing
idempotent. The DUPLICATE becomes harmless, which is the same
observable outcome as exactly-once.

The standard mechanisms:

The transactional outbox is the answer to "how do you write to the database and publish an event atomically?" — a very common follow-up. You cannot two-phase-commit across a database and a broker in practice, so you write both the state change and the event into the same database transaction, and a separate relay reads the outbox table and publishes. The publish is at-least-once, so the consumer still needs to dedupe.

#4 · Consumer groups and scaling

topic with 4 partitions, consumer group of 2:

  P0 ─┐
  P1 ─┴─> consumer A
  P2 ─┐
  P3 ─┴─> consumer B

scale to 4 consumers -> one partition each  (maximum parallelism)
scale to 5 consumers -> one sits IDLE

Partitions are the unit of parallelism, and this caps your consumers. You cannot have more useful consumers than partitions, and repartitioning a live topic is disruptive because it changes key placement — so partition count is a capacity decision made early. Saying this shows operational experience.

The partition key decides ordering. Messages with the same key land on the same partition and are ordered relative to each other. user_id as the key gives per-user ordering, which is almost always what you actually need — global ordering would mean one partition and therefore one consumer.

What happens when a consumer dies is a good deep-dive target:

#5 · Backpressure and lag

Consumer lag is the number to monitor — how far behind the head the consumer is. It is the single best health signal for an async system, and "I'd alert on consumer lag" is a good thing to say unprompted.

lag rising steadily     -> consumers are underprovisioned
lag spiking then draining -> normal; the queue is doing its job
lag rising with idle consumers -> a poison message or a stuck partition

When producers outrun consumers, something must give:

An unbounded queue is not a solution, it is a deferred outage. If arrival rate exceeds service rate persistently, the queue only decides when you fail, not whether. Say this — it is the queueing-theory point most candidates miss, and it reframes "just add a queue" correctly.

#6 · Retries and the dead-letter queue

flowchart LR
    Q["Queue"] --> C["Consumer"]
    C -->|"success"| A["Ack"]
    C -->|"failure"| R{"Retries<br/>exhausted?"}
    R -->|"no"| B["Backoff<br/>+ jitter"]
    B --> Q
    R -->|"yes"| D[["Dead-letter<br/>queue"]]
    D --> H["Alert + human<br/>inspection"]

Exponential backoff with jitter, not fixed retries: delay = base × 2^n ± random. The jitter matters — without it, everything that failed together retries together, and you have rebuilt the thundering herd you were trying to avoid.

A dead-letter queue is mandatory, and its purpose is containment. Without one, a single unprocessable message ("poison pill") is retried forever, blocking its partition and consuming capacity indefinitely. With one, after N attempts it moves aside and everything else proceeds.

Alert on DLQ depth. A DLQ nobody looks at is a silent data-loss channel — it looks healthy precisely because the failures are hidden.

#7 · Choosing a technology

Name a category and one example, then justify by property, not brand. "An append-only log with consumer groups and replay — Kafka, or Redis Streams if we already run Redis and the volume is modest."

#8 · What to say in the round

"The write path is synchronous only as far as durably storing the post. Fan-out goes on a log — I want replay, and I expect a second consumer later for analytics. Partition by author ID so one author's events stay ordered. Delivery is at-least-once, so the fan-out worker dedupes on (tweet_id, follower_id) with a unique constraint. Retries back off with jitter and land in a DLQ after five attempts, and I'd alert on both consumer lag and DLQ depth. The user-visible cost is that a post may take a second or two to reach followers' timelines — I'd write it into the author's own view synchronously so they always see their own post immediately."

That last sentence is the one that gets noticed: it shows you thought about what eventual consistency feels like to a user, not just what it means to the system.

#9 · Interview questions

#Stop condition

You know this block when you can:

1. state the four benefits and five costs of going async,
2. draw the queue-versus-log distinction and pick by replay,
3. explain why exactly-once delivery is impossible but exactly-once processing is not,
4. describe the transactional outbox,
5. say why an unbounded queue is a deferred outage, and
6. name the two metrics you would alert on.