#LLM APIs & model selection

The one sentence: model choice is a procurement decision wearing an engineering costume — the benchmark number is the least important input.

Teams pick a model from a leaderboard, then spend a year discovering that latency, rate limits, licence terms and version stability were what actually mattered.

#1 · Diagram

   WHAT ACTUALLY DECIDES THE CHOICE, in the order it bites

   1. CAN IT DO THE TASK?        your eval set, not a leaderboard
   2. LATENCY                    p95 under YOUR prompt length, not the marketing figure
   3. COST AT YOUR VOLUME        input and output priced differently. Output is dearer
   4. LICENCE / DATA TERMS       where does the data go, may you train on outputs
   5. RATE LIMITS                the ceiling on your growth, not on your demo
   6. VERSION STABILITY          can the model change under you without warning
   7. BENCHMARK SCORE            the number everyone leads with. It is seventh


   HOSTED API                              SELF-HOSTED OPEN WEIGHTS
   ----------                              ------------------------
   per-token cost, no ops                  fixed GPU cost, all the ops
   best frontier quality                   full control of version + data
   rate limits, vendor roadmap             you own uptime, scaling, upgrades
   data leaves your boundary               data stays put
   scales to zero                          idle GPUs still bill

   CROSSOVER: roughly when sustained load keeps a GPU busy.
   Below that, hosted wins on cost as well as effort.

#2 · Design

Basic — the mechanics you are actually buying. Every hosted API exposes roughly the same surface, and the differences matter more than the model:

Intermediate — the cost model people get wrong. Output tokens usually cost several times input tokens. A summariser (long input, short output) and a generator (short input, long output) at the same total token count have very different bills. Estimate them separately:

   monthly cost = requests
                × (input_tokens × input_price + output_tokens × output_price)

Then apply the two multipliers that actually move it: prompt caching on a repeated system prompt or corpus prefix, and batch pricing for anything asynchronous.

Advanced — the licence is the part with teeth. "Open source" is doing a lot of work in most announcements:

Two clauses to read before any architecture discussion: may you use outputs to train another model, and is your data retained or used for training. Both have defaults that surprise people, and both are procurement questions an engineer is expected to raise.

#3 · Flow

graph TD
  A[Task defined] --> B[Build the eval set FIRST]
  B --> C[Shortlist 3-4 candidates<br/>across price tiers]
  C --> D[Run YOUR eval on each]
  D --> E{Any meet quality?}
  E -->|no| F[Task needs decomposition,<br/>better retrieval, or fine-tuning]
  E -->|yes| G[Measure p95 latency<br/>at your real prompt length]
  G --> H[Model cost at projected volume]
  H --> I{Licence and data terms<br/>acceptable?}
  I -->|no| J[Eliminate, regardless of score]
  I -->|yes| K[Check rate limits against<br/>peak, not average]
  K --> L[Pick the CHEAPEST that passes]
  L --> M[Pin the exact version]
  M --> N[Re-run the eval on every<br/>version change]

L is the discipline that saves the most money. The goal is not the best model; it is the cheapest model that passes your bar. Teams routinely serve a frontier model for a task a mid-tier model handles identically, because they never ran the comparison on their own data.

M is the discipline that saves the most debugging. Pin the dated version, never the floating alias. An alias that silently moves under you turns every quality investigation into archaeology.

#4 · UML — the request path and where time goes

sequenceDiagram
    participant C as Client
    participant G as Your gateway
    participant Ch as Prompt cache
    participant P as Provider
    participant M as Model

    C->>G: request
    G->>G: assemble prompt (system + context + user)
    G->>P: streaming call, pinned model version
    P->>Ch: prefix hit?
    Ch-->>P: cached KV for the shared prefix
    Note over Ch,P: A large stable prefix is the single<br/>biggest cost lever on most workloads
    P->>M: prefill (compute-bound)
    M-->>C: first token  <-- the latency users actually judge
    loop decode
        M-->>C: token (bandwidth-bound)
    end
    Note over G: log: model version, token counts,<br/>latency, cache hit. Without these you<br/>cannot attribute cost or explain a regression

#5 · Example

PRICING = {                      # USD per 1M tokens; verify before relying on it
    "frontier":  {"in": 3.00,  "out": 15.00},
    "mid":       {"in": 0.80,  "out": 4.00},
    "small":     {"in": 0.25,  "out": 1.25},
}


def monthly_cost(tier, requests, in_tok, out_tok, cached_frac=0.0, batch=False):
    """Cost at volume, with the two levers that actually move it.

    Input and output are priced differently -- often 4-5x apart -- so a
    summariser and a generator with identical total tokens have very different
    bills. Estimating with a blended rate is the usual mistake.
    """
    p = PRICING[tier]
    # A cached prefix is charged at a large discount; on a long stable system
    # prompt this frequently halves the bill on its own.
    effective_in = in_tok * (1 - cached_frac) + in_tok * cached_frac * 0.1
    cost = requests * (effective_in * p["in"] + out_tok * p["out"]) / 1_000_000
    return cost * (0.5 if batch else 1.0)


def crossover_requests_per_month(tier, in_tok, out_tok, gpu_monthly=1_500):
    """Roughly where self-hosting stops being more expensive.

    Deliberately generous to the hosted side: it ignores the engineering time
    to build and run serving, which is usually the larger number and the one
    that does not appear on any invoice.
    """
    per_request = monthly_cost(tier, 1, in_tok, out_tok)
    return int(gpu_monthly / per_request) if per_request else 0


for tier in PRICING:
    n = crossover_requests_per_month(tier, in_tok=2_000, out_tok=400)
    print(f"{tier:<9} break-even vs one GPU: ~{n:,} requests/month")

frontier  break-even vs one GPU: ~125,000 requests/month
mid       break-even vs one GPU: ~468,750 requests/month
small     break-even vs one GPU: ~1,714,285 requests/month

Read that table honestly. Below roughly a hundred thousand requests a month on a frontier model, self-hosting is not a cost decision — it is a control, latency or data-residency decision, and it should be argued on those grounds.

#6 · Depth — the senior layer

Choosing one model is often the wrong frame. Most production traffic does not need your best model, and the cheapest large win available is usually to send the easy majority somewhere cheaper — a cascade, a big-little pair, or a router on task type. See Ensembles, cascades & routing for the break-even arithmetic, which decides whether that is worth building before you build it.

Benchmark scores are the weakest signal you will be given. Contamination is widespread, benchmarks measure averages over distributions unlike yours, and a two-point MMLU difference predicts nothing about your task. A 50-item eval set built from your own traffic outperforms every leaderboard for this decision. That is also why the evaluation module comes first in this handbook.

Advertised context length is not usable context length. Recall degrades well before the stated limit, and it degrades unevenly — material in the middle of a long context is recovered far less reliably than material at either end. If you plan to use 100k tokens, test retrieval at 100k tokens on your data before designing around it.

Rate limits shape architecture more than model choice does. Limits are usually per-minute on both requests and tokens, and they bind at peak, not average. A workload with a 10× daily peak needs headroom for the peak. Design a queue with backpressure from the start; discovering the limit in production means discovering it as a user-visible outage.

Multi-provider is worth less than it sounds and more than nothing. A thin interface plus a tested fallback is cheap insurance against an outage. But prompts do not transfer cleanly between model families, so a true hot-swap needs its own eval run per provider. Budget for that rather than assuming portability.

Self-hosting's real cost is not the GPU. It is serving expertise, on-call, capacity planning, upgrades, and a quantization pipeline. Those costs are invisible on a spreadsheet and dominant in practice. Self-host for control, data residency, or genuinely high sustained volume — not because the per-token arithmetic looked good.

#7 · From each seat

#8 · Interview questions

#Stop condition

You are done when you can:

1. list the seven selection criteria and explain why benchmarks come last,
2. do the hosted-versus-self-hosted break-even arithmetic,
3. name the two licence clauses that matter and why,
4. explain why output tokens dominate a generation workload's bill, and
5. say what pinning a version protects you from.

#Sources worth reading