#Numbers to know

In a system design round nobody hands you a calculator. What separates a convincing answer from a vague one is being able to say "that's about 14 gigabytes, so it won't fit" without pausing.

Everything here is approximate on purpose. The point is the order of magnitude and the shape of the trade-off — being able to reason, not to quote.

#Model memory

The one formula: bytes = parameters × bytes_per_parameter

Add 15–20% for activations and framework overhead, plus the KV cache below.

The shortcut worth memorising: at FP16, gigabytes ≈ 2 × billions of parameters. At INT4, gigabytes ≈ half the billions. A 70B at INT4 is ~35 GB and fits on one 40 GB card; at FP16 it does not fit on two.

#KV cache

   bytes = 2 × layers × kv_heads × head_dim × seq_len × batch × bytes_per_value

For a Llama-3-8B shape (32 layers, 8 KV heads with GQA, head_dim 128, FP16):

Read the diagonal. At long context and real batch sizes the cache dwarfs the 16 GB of weights. This is why context length and concurrency trade directly against each other, and why "we support 128k" is a capacity statement.

#Latency

The decode ceiling from bandwidth alone, batch 1:

   tokens/sec ≈ memory_bandwidth / model_bytes

   7B @ FP16 on 2 TB/s   =  2000 / 14   ≈ 143 tok/s
   7B @ INT4 on 2 TB/s   =  2000 / 3.5  ≈ 571 tok/s

If your measured rate is far below the ceiling, quantization is not your problem — look at batching, queueing or the framework.

#Tokens

100k words ≈ 133k tokens. Useful for "will this book fit in the context window" questions, and the answer is usually no.

#Cost

Illustrative tiers per million tokens — verify current prices before quoting them, they move downward regularly:

   cost per request = (in_tokens × in_price + out_tokens × out_price) / 1e6

A typical RAG request — 2,000 in, 400 out, frontier tier:

   (2000 × 3 + 400 × 15) / 1e6  =  $0.012

Output is roughly 5× the price of input. A summariser and a generator with the same total token count have very different bills, so estimate them separately.

Break-even against one GPU (~$1,500/month) at that request shape: roughly 125,000 requests/month. Below that, self-hosting is not a cost decision.

#Retrieval & evaluation

Confidence intervals on a pass rate — the table that stops you shipping noise:

On 50 items, an improvement from 80% to 84% is indistinguishable from noise. If you remember one line from this page, this is the one — it is the most common statistical error in LLM engineering, and saying it out loud is a strong senior signal.

Judge agreement (Cohen's kappa):

#Agents

Reliability compounds. At p success per step:

This single table explains agent design. Long autonomous chains do not fail because models are weak; they fail because reliability multiplies. Every mitigation — checkpoints, decomposition, human approval — is a way of shortening the chain.

#Fine-tuning

#Using this on a whiteboard

The five that come up most, in the form you should be able to produce cold:

1. "Will it fit?" → params × bytes/param, plus 20%, plus the KV cache.
2. "How fast can it go?" → bandwidth ÷ model bytes = tokens/sec ceiling.
3. "What will it cost?" → tokens × price, input and output separately.
4. "Is that improvement real?" → check n against the interval table.
5. "Will the agent work?" → p per step, raised to the number of steps.

Being able to do these in your head is disproportionately convincing, because it demonstrates you have actually built something rather than read about it.