A chat model does two different jobs for every request. First it reads the whole prompt in one heavy burst, which produces the first word of the reply. Then it writes the rest one word at a time, in quick small steps shared with every other reply in progress. Run both jobs on the same GPU and each long prompt that arrives freezes every reply being written there.
Every arrangement sees exactly the same requests. Click a row to watch it.
Same GPU, whole prompts: the orange blocks are iterations that read a prompt, and while one runs, every reply on that GPU waits. A long prompt freezes them for about 0.4 seconds, so at the default load almost every reply freezes at least once: only 1% are on time. Yet the median gap between words is 41 ms, inside the 50 ms target. The average hides the freezes.
Prompts in chunks reads at most 512 prompt tokens per iteration, so no freeze is longer than about 60 ms. The price is that prompts take longer to read, because they share every iteration with the writing: the slowest tenth wait over a second for their first word, and 89% are on time.
Separate GPUs: two GPUs only read prompts and two only write replies, so writing never stops for reading: 100% on time. The one visible pause, about 0.1 s, is right after the first word, while the prompt's memory is shipped to the writing GPU. Try one reading GPU, or three: either side runs short, and on-time drops to about 10%. Splitting only helps if the split fits the traffic.
Each GPU works in iterations. An iteration writes one word for every reply in progress and may also read prompt tokens: 12 ms, plus 0.4 ms per reply, plus 0.05 ms per prompt token read. Waiting prompts go to the reading GPU with the least work queued, and replies to the writing GPU with the fewest replies.
Three requests in ten have a long prompt (3,000–8,000 tokens), the rest 200–1,200; replies are 100–400 words. Shipping a prompt's memory (its KV cache) to another GPU takes 0.01 ms per prompt token.
A reply is on time if its first word comes within 2 s, its average gap is within 50 ms, and it never freezes for more than 250 ms. The model charges chunking only the fixed cost of extra iterations; real chunked reading also re-reads earlier chunks' memory, so it flatters chunking a little.
The simulation is Go (sims/disagg), compiled to WebAssembly and run in your browser; go run ./cmd/mini disagg prints the same comparison in a terminal. Its tests check the claims on this page.