Here's a detail about modern AI serving that surprises most engineers: when you send a prompt to a frontier LLM, the machine that reads your prompt and the machine that writes the answer are increasingly not the same computer.
This isn't an implementation quirk. It's the biggest architectural shift in AI infrastructure right now — and underneath it is a distributed-systems story every backend engineer already knows.
Two phases, opposite personalities
Generating an LLM response has two phases with completely different resource profiles:
- Prefill reads your entire prompt in one parallel pass. Thousands of tokens hit the GPU at once — it's compute-bound, saturating the chip's arithmetic units while barely touching memory bandwidth.
- Decode writes the answer one token at a time. Every single token requires re-reading the model weights and all the attention state so far — it's memory-bandwidth-bound, hammering HBM while the compute units mostly idle.
And decode is where the time goes: a 1,000-token reply at ~20 ms/token is about 20 seconds of decode after just 1–2 seconds of prefill.
Put both phases on the same GPU — how everyone served LLMs until recently — and you get a machine that is always wasting one of its two most expensive resources. Worse, a heavy prefill landing on a box that's mid-decode for other users visibly stutters their responses.
So the industry did what distributed-systems people always do when one box hosts two workloads with opposite bottlenecks: split them apart and specialize.
The KV cache is the new hot data
If prefill and decode run on different machines, the attention state computed during prefill — the KV cache — has to travel between them. That state is not small: for a 70B model in BF16, it's roughly 320 KB per token, so a 32K-token context is about 10 GB per user.
The moment that cache goes over a wire, it stops being an internal GPU buffer and becomes what we'd call data. And every classic storage question comes rushing in:
- Tiering — hot cache in GPU HBM, warm in DRAM, cold on SSD, shipped over RDMA. Mooncake, the system behind the Kimi assistant (and FAST '25 best paper), pools CPU, DRAM, and SSD across the whole cluster into one cache — and served 59–498% more requests on real production traces.
- Working sets — a USENIX ATC '25 study of Alibaba Cloud traffic found 10% of cache blocks account for 77% of reuses, with a P99 cache lifespan of just 97 seconds. Hot, short-lived, highly skewed: a textbook caching workload.
- Cache hits that aren't yours — the same study found 97% of KV reuse in API traffic is single-turn: millions of different users hitting the same 4,000-token system prompt. Cache it once, and everyone's time-to-first-token drops. LMCache reports 3–10× faster first tokens and up to 15× throughput on prefix-heavy workloads.
The economics seal it: an H100 hour costs $4–6 in 2026; the networked storage to hold cache costs fractions of a cent. Storing attention is now dramatically cheaper than recomputing it. Memcached taught us this twenty years ago — it's just that the "database query" being cached is now a matrix multiplication.
This is now the default architecture
This stopped being research in March: NVIDIA Dynamo 1.0 went GA at GTC 2026 with disaggregated prefill/decode as a first-class deployment shape — separate worker pools, a routing layer, and a transfer library (NIXL) that moves cache across NVLink, InfiniBand, and Ethernet. NVIDIA reports up to 7× throughput on DeepSeek-R1 on Blackwell. vLLM, SGLang — every major serving stack now ships it. There's even a 2026 paper proposing prefill-as-a-service, with KV cache shipped across datacenters.
Why this sparks my curiosity
The lesson I keep coming back to: the bottleneck decides the architecture. Web serving split into app tiers and cache tiers when we noticed reads and writes behave differently. Databases split storage from compute when the cloud made them scale differently. LLM inference just hit the same fork in the road — and it's rediscovering tiered caches, working sets, eviction policies, and wire formats, one FAST paper at a time.
If you're a backend engineer feeling like AI infra is someone else's field: it isn't. The GPUs are new. The problems are yours.
Sources: Why LLM inference is disaggregating its memory (S. Seshadri) · Mooncake: KVCache-centric disaggregated serving (FAST '25) · NVIDIA Dynamo · KVCache Cache in the Wild (USENIX ATC '25)
Somoprovo, by firelight 🏕️