#Long context

The one sentence: the advertised context length is what the model will accept, not what it will reliably use — and the gap between those is where most "we don't need RAG any more" projects fail.

Every context-window expansion revives the claim that retrieval is obsolete. The claim is testable, the test is cheap, and it has so far come out the same way each time.

#1 · Diagram

   WHAT ACTUALLY HAPPENS AS CONTEXT GROWS

   recall of a fact placed at position X in a long context

   1.0 ┤███                                              ███
       │███                                              ███
   0.8 ┤███                                              ███
       │  ███                                          ███
   0.6 ┤    ███                                      ███
       │       ████                              ████
   0.4 ┤           ██████                  ██████
       │                 ██████████████████
   0.2 ┤
       └────┬──────────┬──────────┬──────────┬──────────┬────
          start                 middle                  end

   THE "LOST IN THE MIDDLE" SHAPE. Beginning and end are recovered well;
   the middle is not. And the dip DEEPENS as the context grows.


   THE THREE COSTS OF A LONG PROMPT, all paid on every request

   MONEY      every token billed, every time
   LATENCY    prefill is quadratic-ish -- TTFT grows with prompt length
   MEMORY     KV cache grows linearly with context AND batch

#2 · Design

Intermediate — how long context is achieved. Models are rarely trained at their full advertised length; that would be prohibitively expensive. Instead:

This is why quality degrades in the extended region. Positions beyond the original training length were interpolated into, not learned from data. A model advertised at 128k may have been trained at 8k and stretched. It will accept 128k tokens; whether it uses them well is an empirical question about that specific model.

Those four are context extension — stretching a model past what it was trained on. Worth separating from architectural long context: extension takes a checkpoint and reinterprets its positions, so the quality question is always "how far past its training length has this been pushed", while an architecture built for length has a different failure mode entirely.

Advanced — measure it yourself, because vendor claims are about acceptance. The standard probe is needle in a haystack: place a specific fact at varying depths in contexts of varying lengths, then ask for it. It gives a two-dimensional map of recall by (length, position).

Its limitation matters too: retrieving one distinctive sentence is much easier than reasoning over dispersed information. A model can score perfectly on needle-in-a-haystack and still fail to compare three facts scattered through the same document. Test with the multi-fact variant if that is your use case.

"Infinite context" is a claim about memory, not about recall. The architectures marketed this way — compressive-memory schemes that fold older tokens into a fixed-size state alongside a local attention window, streaming caches that keep attention sinks plus a rolling window — all share one property: bounded state. They will accept an unbounded stream without running out of memory, which is genuinely useful for a long-running agent or a transcript.

What they do not offer is unbounded recall. Once a token has been folded into a fixed-size summary it cannot be recovered verbatim, so the failure mode is a model that keeps answering fluently about material it can no longer actually see. Read "infinite context" as "never OOMs", and then probe recall separately — it is a different claim and it is the one you were buying.

Shallow prefill is the newer idea worth knowing, because it exploits an asymmetry nobody used for years: deeper layers hold progressively more redundant KV, and upper-layer attention contributes less to gathering prefill information than lower-layer attention does. So run prefill over fewer layers than decode — shallow prefill, deep decoding. It attacks all three coupled long-context costs at once, because prefill depth drives TTFT, and the KV you never computed is KV you never store or read.

Note how it composes with the rest of this page: sparsity cuts prefill width (which keys), shallow prefill cuts prefill depth (which layers). Different terms, so they multiply.

#3 · Flow — long context or retrieval?

graph TD
  A[Corpus to answer over] --> B{Total size?}
  B -->|fits comfortably in context| C{Query volume?}
  B -->|larger than context| D[Retrieval. No choice]
  C -->|low, exploratory| E[Long context is simpler.<br/>Use it]
  C -->|high| F{Same corpus every request?}
  F -->|yes| G[Long context + PREFIX CACHING<br/>can beat retrieval on simplicity]
  F -->|no| H[Retrieval. You are paying for<br/>the whole corpus per query]
  D --> I[Chunk, embed, retrieve, rerank]
  E --> J[Measure recall at YOUR length]
  G --> J
  I --> J
  J --> K{Recall acceptable?}
  K -->|no| L[Retrieval, or hybrid:<br/>retrieve, then long context over the results]
  K -->|yes| M[Ship]

Branch G is the genuinely interesting case. If every request shares the same large prefix, prompt caching makes long context far cheaper than the naive arithmetic suggests — sometimes cheaper than running a retrieval stack. That is a real architectural option, not a compromise, and it is under-considered.

Branch L is the answer that usually wins: retrieval to narrow from millions of tokens to tens of thousands, then long context to reason over what came back. They compose. The framing of "RAG versus long context" is a false choice.

#4 · UML — where the cost lands

sequenceDiagram
    participant C as Client
    participant G as Gateway
    participant Ca as Prefix cache
    participant M as Model

    C->>G: query + 100k-token corpus
    G->>M: prompt
    M->>Ca: is the prefix cached?
    alt cache hit (same corpus as last time)
        Ca-->>M: reuse KV for the shared prefix
        Note over Ca,M: Large discount on input tokens,<br/>and TTFT drops sharply
    else cache miss
        M->>M: PREFILL all 100k tokens
        Note over M: Seconds of compute.<br/>This is the TTFT you feel.
    end
    M-->>C: first token
    loop decode
        Note over M: KV cache now holds 100k positions --<br/>this is what limits batch size
        M-->>C: token
    end

#5 · Example

def context_cost(tokens_in: int, tokens_out: int, in_price=3.0, out_price=15.0,
                 cached_fraction=0.0, cache_discount=0.1):
    """Cost of a request, with prefix caching accounted for.

    Caching is what makes long context economically viable for a fixed corpus.
    Without it, you pay for the entire corpus on every single query, which is
    the arithmetic that usually decides the question.
    """
    uncached = tokens_in * (1 - cached_fraction)
    cached = tokens_in * cached_fraction * cache_discount
    return ((uncached + cached) * in_price + tokens_out * out_price) / 1e6


corpus = 100_000
print(f"no cache : ${context_cost(corpus, 500):.3f}/query")
print(f"90% cached: ${context_cost(corpus, 500, cached_fraction=0.9):.3f}/query")
print(f"RAG (5 chunks, ~2.5k tokens): ${context_cost(2_500, 500):.3f}/query")

no cache  : $0.308/query          <- 100k tokens, every time
90% cached: $0.038/query          <- caching changes the answer
RAG (5 chunks, ~2.5k tokens): $0.015/query

The grid above is the one you should run. Before you do, here is the grid your own assumptions already imply — state the recall curve you believe in and watch it decide, at every (length, depth), what a needle test would have reported.

The measurement that should decide it:

def needle_test(model, filler_corpus, needle, question, depths, lengths):
    """Recall of a known fact by (context length, position).

    Run this on YOUR model at YOUR lengths before designing around a big
    window. Vendor claims describe what the model ACCEPTS. This measures what
    it uses -- and the two diverge, usually in the middle.
    """
    results = {}
    for length in lengths:
        for depth in depths:                       # 0.0 = start, 1.0 = end
            context = _insert_at(filler_corpus[:length], needle, depth)
            answer = model.ask(context, question)
            results[(length, depth)] = _contains_answer(answer, needle)
    return results

Run it once. The resulting grid is worth more than any published benchmark, because it is your model, your document style, and your question type.

#6 · Depth — the senior layer

Long context does not remove the need for retrieval; it moves where the selection happens. With a 200k window and a 50-million-token corpus, you still must choose what goes in. The only question is whether that choice is made by a retriever you can measure and debug, or implicitly by whatever truncation rule your code happens to apply.

Attribution and debuggability are the under-rated columns. When a RAG system answers wrongly you can look at what it retrieved. When a long-context system answers wrongly you have a 100k-token prompt and a shrug.

Position the important material at the ends. Given the recall shape, put the instruction last and the most relevant retrieved chunks first and last, not buried in the middle. This is nearly free and measurably helps — and it is a concrete, unusual thing to say in an interview.

Where long context genuinely wins: a single large document that must be reasoned over as a whole — a contract, a codebase module, a long transcript. Chunking actively destroys the cross-references that make those documents coherent. That is the case where retrieval is the wrong tool and the window is the right one.

#7 · From each seat

#8 · Interview questions

#Stop condition

You are done when you can:

1. draw the lost-in-the-middle recall shape,
2. explain why extended context degrades in terms of position interpolation,
3. give the cost, latency and memory consequences of a long prompt,
4. argue the retrieve-then-reason composition rather than the either/or, and
5. name the case where long context is clearly the right tool.

#Sources worth reading