#Decoding & sampling

The one sentence: the model does not produce text — it produces a probability distribution over the vocabulary, once per token, and decoding is the policy you apply to turn that into a choice.

Two systems with the same model and the same prompt can behave completely differently because of settings most people copy from a tutorial without reading.

#1 · Diagram

   ONE STEP OF GENERATION

   logits over the full vocabulary (~100k numbers)
        │
        ├── temperature ──► sharpen (T<1) or flatten (T>1) the distribution
        │
        ├── top-k ────────► keep only the k most likely
        │
        ├── top-p ────────► keep the smallest set summing to p  (nucleus)
        │
        ├── penalties ────► push down tokens already used
        │
        ▼
   renormalise ──► sample one token ──► append ──► repeat


   THE SHAPE OF THE CHOICE

   greedy (T=0)         always the top token. Repetitive, but reproducible-ish
   low T (0.1-0.3)      focused. Extraction, classification, code
   mid T (0.7-1.0)      natural. Chat, general use
   high T (>1.2)        erratic. Rarely what you want

#2 · Design

Basic — the parameters and what each actually does.

Top-p beats top-k, and the reason is worth internalising. Top-k always keeps exactly k candidates regardless of how confident the model is. When it is certain — the next token is obviously "(" — top-k still admits 39 wrong alternatives. When it is genuinely uncertain across 200 plausible continuations, top-k truncates to 40 and discards good options. Top-p adapts: it keeps few candidates when confident and many when not, which is the behaviour you actually want. Use top-p; leave top-k off unless you have a specific reason.

Intermediate — beam search, and why chat models do not use it. Beam search keeps the b highest-probability partial sequences and returns the most likely complete one. It genuinely helps where a single correct output exists — translation, structured extraction. For open-ended generation it produces bland, repetitive text, because the highest-probability sequence is the least surprising one. Human-sounding text is not maximum-likelihood text, which is a genuinely interesting finding and the reason sampling won for chat.

Advanced — temperature 0 is not determinism, and this matters more than it sounds. Even at temperature 0 you can get different outputs from identical inputs. Causes:

1. Floating-point non-associativity — GPU reductions sum in nondeterministic order, so near-ties in logits can flip.
2. Batch composition — kernels select different code paths by batch shape, changing rounding.
3. Mixture-of-experts routing — expert assignment can depend on other sequences in the batch.
4. Silent infrastructure changes — different GPU generation, different kernel version.

So "temperature 0 for reproducibility" is a reduction in variance, not a guarantee. Any test asserting exact string equality on model output is a flaky test waiting to happen — which is precisely why the evaluation module scores rather than asserts.

#2b · The rest of the decoding algorithms

§2 covered the samplers you will actually configure. Taxonomies list a dozen more, and they sort cleanly into three groups — which is more useful than the list, because the groups tell you whether a given name is a sampler, a kernel, or a different generation paradigm.

#Still samplers — they change which token is drawn

• Min-p sampling cuts at a threshold relative to the top token's probability — keep everything above p × p_max — rather than at a fixed cumulative mass. The effect is that a confident step keeps almost nothing and an uncertain one keeps a lot, automatically. It is the most defensible alternative to top-p, and the argument for it is that top_p = 0.95 means something very different when the top token has 0.9 than when it has 0.15.
• Contrastive decoding scores with two models — a strong one and a deliberately weak one — and prefers tokens the strong model likes and the weak one does not. The weak model acts as a proxy for "generic, degenerate continuation", so subtracting it suppresses repetition without a penalty term. The cost is a second forward pass per step.

Both sit at the same place in the pipeline as temperature and top-p: after the logits, before the draw. Everything in decoding §2 composes with them.

#Not samplers — they change how the same tokens are computed

• Flash decoding is a kernel, despite the name's symmetry with FlashAttention. At decode time there is exactly one query and a very long KV cache, so the usual parallelisation over query positions has nothing to work with and the GPU sits mostly idle. Flash decoding splits the key/value dimension across thread blocks instead and combines the partial softmax results — the same online-softmax trick, applied along a different axis. It changes no output at all, which is the tell that it belongs on the kernel page rather than this one.

#Different paradigms — they change how many tokens come out

• Non-autoregressive decoding generates the whole sequence in parallel rather than one token at a time. It is genuinely fast and generally worse: without conditioning on what it just emitted, the model cannot keep a long output self-consistent. It survives in translation and speech, not in open-ended generation.
• Multi-token generation (multi-token prediction) is the practical middle: train the model with extra heads that predict positions t+1, t+2, …, then use those extra predictions as a free draft for speculative decoding. The model becomes its own draft model, which removes the awkward part of speculation — finding a small model that agrees with the big one.

The distinction worth carrying: a sampler changes what you get, a kernel changes how fast you get it, and a paradigm changes what "a step" means. Names like flash decoding and non-autoregressive decoding sit in the same list on every taxonomy and belong to three different layers of the stack.

#3 · Flow

graph TD
  A[Task] --> B{One correct answer,<br/>or many acceptable ones?}
  B -->|one| C[temperature 0<br/>extraction, classification, code]
  B -->|many| D[temperature 0.7-1.0 + top_p 0.95<br/>chat, drafting, ideation]
  C --> E{Output must be structured?}
  E -->|yes| F[Add constrained decoding<br/>or schema-enforced tools]
  E -->|no| G[Set stop sequences]
  D --> H{Repetitive output?}
  H -->|yes| I[Raise presence penalty<br/>before raising temperature]
  H -->|no| G
  F --> J[Always set max_tokens]
  G --> J
  I --> J

I is a common misdiagnosis. Repetition loops are usually better fixed with a presence or frequency penalty than by raising temperature — raising temperature fixes the loop by making everything less predictable, including the parts that were fine.

#4 · UML — where sampling sits in the loop

sequenceDiagram
    participant M as Model
    participant S as Sampler
    participant G as Grammar (optional)
    participant O as Output

    loop each token
        M-->>S: logits over vocabulary
        opt constrained decoding
            G-->>S: mask invalid tokens to -inf
            Note over G,S: This is why constrained output<br/>cannot produce invalid JSON:<br/>invalid tokens are unsamplable
        end
        S->>S: temperature, top-p, penalties
        S->>S: renormalise, sample
        S-->>O: token
        O-->>M: append and continue
    end

Constrained decoding is a mask applied to the logits, not a validation step afterwards. That is why its guarantee is absolute rather than probabilistic.

The code above is the order the operations must happen in; the lab below is that order running, with the filters fighting over twelve fixed logits so you can see exactly which token each one takes.

#5 · Example

import numpy as np


def sample(logits, temperature=1.0, top_p=0.95, penalties=None):
    """One decoding step, in the order the operations must happen.

    Penalties are applied to raw logits BEFORE temperature: applying them after
    scaling makes their strength depend on the temperature, so tuning one
    silently changes the other.
    """
    logits = logits.astype(np.float64).copy()

    if penalties:
        for token_id, count in penalties.items():
            logits[token_id] -= count       # frequency-style: proportional to use

    if temperature <= 0:
        return int(np.argmax(logits))       # greedy; no sampling at all

    logits /= temperature
    probs = np.exp(logits - logits.max())
    probs /= probs.sum()

    if top_p < 1.0:
        order = np.argsort(probs)[::-1]
        cumulative = np.cumsum(probs[order])
        # Keep everything up to and INCLUDING the token that crosses p, so the
        # nucleus is never empty when one token already exceeds p on its own.
        cutoff = int(np.searchsorted(cumulative, top_p) + 1)
        keep = order[:cutoff]
        filtered = np.zeros_like(probs)
        filtered[keep] = probs[keep]
        probs = filtered / filtered.sum()

    return int(np.random.choice(len(probs), p=probs))

Settings by task, as a table you can act on:

PRESETS = {
    # One right answer: remove variance entirely.
    "extraction":     {"temperature": 0.0, "max_tokens": 500},
    "classification": {"temperature": 0.0, "max_tokens": 10},
    "code":           {"temperature": 0.2, "top_p": 0.95, "max_tokens": 2000},

    # Many acceptable answers: sample.
    "chat":           {"temperature": 0.7, "top_p": 0.95, "max_tokens": 1000},
    "brainstorm":     {"temperature": 1.0, "top_p": 0.98, "max_tokens": 800},

    # Judges must be as reproducible as possible -- a ruler that varies run to
    # run cannot measure a change.
    "judge":          {"temperature": 0.0, "max_tokens": 200},
}

#6 · Depth — the senior layer

Structured output and sampling interact, and the interaction is easy to get wrong. High temperature raises the chance of a malformed JSON token. But clamping to temperature 0 and a tight grammar can hurt reasoning quality, because the model cannot express intermediate thinking. The pattern that works: low temperature, a free-text reasoning field first, then constrained fields — {"reasoning": "...", "answer": "..."} in that order.

Penalties are blunt instruments applied to token ids, not meanings. A frequency penalty suppresses a token regardless of whether repeating it is correct. In code generation this is actively harmful — return, self and def are supposed to repeat. Penalties on code, or on structured output, cause subtle corruption. Default to 0 and reach for them only for a demonstrated repetition loop.

Speculative decoding is one member of a family, and the others are worth recognising because they trade the draft model for something cheaper:

What they all share is the property that makes speculative decoding safe in the first place: the large model verifies, so the output distribution is unchanged. Guessing badly costs throughput, never correctness — which is why this family is unusually safe to deploy and why lookup decoding, requiring no model at all, is worth trying before anything with a draft model in it.

The one to reach for first is prompt lookup: on any task where the output substantially quotes the input, copying n-grams from the prompt is a near-free 2–3×, and it needs nothing but a dictionary.

Speculative decoding deserves a mention here because it is a decoding-time optimisation that changes nothing about the output distribution. A small draft model proposes several tokens; the large model verifies them in one forward pass and accepts the longest correct prefix. Because verification is parallel and decode is bandwidth-bound, this is often a two-to-three-times speedup with mathematically identical output. That last property is what makes it safe to deploy without re-running your evals — worth saying explicitly, because it is unusual among optimisations.

FR-Spec is the refinement worth knowing, because it fixes a bottleneck that only appears at modern vocabulary sizes. Methods like EAGLE compress the draft model down to roughly one layer plus an LM head — at which point, with a 128k vocabulary, the LM head is most of the draft cost. FR-Spec shrinks the draft's vocabulary to a frequency-ranked shortlist, exploiting the long tail of token frequency: a small high-frequency subset covers most of what a drafter proposes. Reported: LM-head compute down about 75%, and roughly 1.12× over EAGLE-2. The property that makes it safe is the same one that makes speculative decoding safe — verification still runs over the full vocabulary, so the output distribution is unchanged. The draft is allowed to be narrow because the draft is only ever a guess.

Logprobs are underused. Most APIs can return per-token log-probabilities, which give you a free confidence signal: low mean logprob on a generated answer correlates with the model being unsure. It is not calibrated and should not be presented to users as a probability, but as a routing signal — escalate low confidence to a larger model or to a human — it costs nothing and works.

#7 · From each seat

#8 · Interview questions

#Stop condition

You are done when you can:

1. explain temperature, top-k and top-p and why top-p is preferred,
2. say why temperature 0 is not determinism and name a cause,
3. explain why beam search lost for chat but not for translation,
4. give the right settings for extraction, chat and judging, and
5. describe speculative decoding and why it needs no re-evaluation.

#Sources worth reading