A GPU writes replies one word-piece (token) at a time, but it can move eight replies forward in one step for barely more than the cost of one. So it works in batches. The question is when a new request gets a seat: when the whole batch is done, or as soon as any one reply finishes.
Exactly the same requests arrive at the same moments under both. Click a row to watch it.
Static batching seats up to eight requests and runs them until the longest reply is done. The short replies finish in a second or two and their seats sit empty, shown hatched red, while new requests pile up in the waiting row. At the default settings half of all seat-time is wasted this way, and the average wait for a first token is 16.4 seconds.
Continuous batching checks the queue before every step and fills any empty seat at once. It never leaves a seat empty while someone is waiting. The average wait for a first token drops to 0.4 seconds and replies finish in 3.7 seconds on average instead of 19.3.
Raise the load to 2 requests a second and the gap shows up in raw output too: about 181 tokens a second against 103. Make reply lengths more even and static batching wastes less, because a batch's replies finish closer together.
One GPU with eight seats. Each step writes one token for every reply in the batch and takes 20 ms plus 2 ms per reply. A request that joins also has its prompt (200–600 tokens) read in that step, at 20,000 tokens a second.
Reply lengths are log-normal around a median of 80 tokens: most are short, a few run past 500, clipped at 1,200. Requests arrive at random (a Poisson stream) for 60 seconds; the run ends when the last reply is written.
The idea is iteration-level scheduling from Yu et al., Orca: A Distributed Serving System for Transformer-Based Generative Models (OSDI 2022), now standard in LLM servers.
The simulation is Go (sims/batching), compiled to WebAssembly and run in your browser; go run ./cmd/mini batching prints the same comparison in a terminal. Its tests check the claims on this page.