Most requests to a chat service begin with the same long prompt: a system prompt, a set of examples, a document. A GPU that has already read that prompt keeps the result and can skip straight to the new part. So where you send the next request decides how much work gets done twice.
Every policy sees exactly the same requests. Click a row to watch it.
Round robin and least loaded spread the work evenly but scatter each prompt across all eight GPUs, so most requests arrive at a GPU that has never seen their prompt and has to read it from scratch. The queues fill with recomputation.
Cache affinity sends every request with the same prompt to the same GPU, so the prompt is almost always cached. But the most popular app's GPU now does far more than its share. Watch one lane stay busy while others idle.
Affinity with a load cap keeps the affinity until a GPU gets too far ahead of the average, then spills the overflow to the next GPU on the ring, which caches that prompt too. Hot prompts end up on two or three GPUs; cold ones on one.
Each GPU reads prompts one at a time at 10,000 tokens a second; writing the reply runs alongside and doesn't hold up the queue. The wait for the first token is time in the queue plus reading whatever part of the prompt isn't cached.
Each GPU caches whole prompts, dropping the least recently used when full. Prompt popularity follows a Zipf law: a few apps send most of the traffic, which is what makes plain affinity lopsided.
The load cap is consistent hashing with bounded loads (Mirrokni, Thorup and Zadimoghaddam, 2018), measured in queued seconds rather than request counts.
The simulation is Go (sims/cacherouter), compiled to WebAssembly and run in your browser; go run ./cmd/mini cache-router prints the same comparison in a terminal. Its tests check the claims on this page.