Training LLMs · 4

The one slow GPU

Training a large model splits every step across thousands of GPUs, and each step ends only when the last GPU has finished and they have all shared their results. So one slow GPU holds up every other one. And with thousands of GPUs, something somewhere fails every few hours, throwing the run back to its last saved checkpoint.

Underneath: waiting for the slowest of many; the Young–Daly checkpoint intervalSeen in: every large synchronous training run. Meta reported 419 unexpected interruptions in 54 days of training Llama 3 on 16,384 GPUs

One step at a time.

Two weeks of training, one row per day. Green is training that was kept; red hatching is training thrown away by a failure; blue slivers are checkpoints being written.

Failures

–

so far

Training thrown away

–

redone after failures

Writing checkpoints

–

of the time so far

Time kept

–

training that survived, so far

How often to checkpoint. Share of time that ends up as kept training, for every interval (failures and checkpoints only, from the exact expected value). The dashed line is the Young–Daly interval; the dot is your setting.

Five ways to run the same two weeks

Every row sees the same failures at the same moments. Goodput is the share of GPU time that becomes kept training: time kept × step efficiency. Click a row's settings to apply them.

What to notice

Scale makes a rare problem constant. A GPU here is slow on one step in 10,000. On 8 GPUs that almost never matters: steps run at 97.5% of their ideal speed. On 16,384 GPUs, 81% of steps wait for at least one slow GPU, the average step takes 1.54 s instead of 1.10 s, and step efficiency falls to 71%.

Backup GPUs run 2% extra and close the step as soon as enough have reported, so the slow few are never waited for. At 16,384 GPUs that lifts step efficiency to 90%. On 8 GPUs one extra GPU costs more than it saves (87% against 97.5%).

Checkpoint interval has a sweet spot. With a failure every 3.1 hours, checkpointing every 3 minutes spends 23% of the time writing checkpoints, while every 2 hours throws away 29% to failures. The Young–Daly interval, about 19 minutes, keeps 87.5% of the time.

How the model works

Each step every GPU computes for 1 s plus a small random delay (1% on average), and with a chance of 1 in 10,000 runs 1.5× slower that step. Then they synchronise for 0.1 s. The slowest GPU of 16,384 is drawn directly from the distribution of the maximum, so every step is a few arithmetic operations at any scale. The step view shows 192 GPUs; every slow GPU and the slowest healthy one are always among them.

Failures are random, with the cluster's mean time between failures set from Meta's figure (The Llama 3 Herd of Models, 2024): 419 in 54 days on 16,384 GPUs, one every 3.1 hours, and proportionally rarer on smaller clusters. A failure loses everything since the last checkpoint plus a 5-minute restart; a checkpoint takes 1 minute. Young and Daly's rule says checkpoint every √(2 × checkpoint cost × time between failures); the exact best here is 18.6 minutes against Young–Daly's 19.3.

Backup workers follow Chen et al., Revisiting Distributed Synchronous SGD (2016). The extra GPUs also fail, and are charged as cost.

The simulation is Go (sims/straggler), compiled to WebAssembly and run in your browser; go run ./cmd/mini straggler prints the same comparison in a terminal. Its tests check the claims on this page, including that the simulation matches the exact expected value over long runs.