The unattended harness

Four pieces that let an experiment run with nobody at the desk. The loop owns execution. A person owns judgement. That split is the design, not a limitation.

What it cost to not have one

Every piece below exists because of a specific loss, and none of those losses was research:

what happened	cost
Two jobs sat in a backend's `RECEIVED` state, never queued and never erroring. No exception was raised, so the retry logic never fired.	144 hours
A backoff tuned for random flakiness, against a failure that recovered on the next attempt every time.	29% of a run, asleep
A gate that measured lift on the injected target — a quantity a logit nudge also moves.	2 hours, and a withdrawn headline
A partial cache read as complete; a cache key missing a parameter that changes the number; a flaky node misread as an oversized batch.	four re-runs

The pieces

preflight — gates that refuse, not warn: Lint, import, cache-key consistency, cache completeness, the run's own reporting path against known-answer data, and one real round trip — cheapest first, before anything touches the network. A gate that warns is a gate nobody reads at 3am.
triage — every failure by cause; UNSORTED is the output: Each warning matched to a known failure mode, carrying the record entry that diagnosed it, so a line at 3am points at the decision that explains it. A known failure needs a count; an unknown one needs a person.
report — one page per run, readable mid-run: Built only from what the run left on disk, so a run still going renders with its missing pieces marked. What needs a human eye goes first, then the ticket's pre-stated outcomes beside the verdict that actually fired.
watch — the loop, and its four alarms: Preflight, run, triage, report, next. It escalates on a refused gate, an unsorted failure, a verdict outside what was expected, or a job outliving its budget. Everything else is logged and the queue carries on.

The boundary

The loop does not choose what to run next, judge whether a result is real, write a record entry, close a ticket, or touch the write-up. Three reasons, all from the project's own record rather than from principle:

A meta-analysis of the project's seventeen results found that every narrowing came from choosing which held-fixed dimension to vary.That choice is judgement, and it is the step that has reliably produced results. A loop picking its own next experiment would be guessing at the one move that works.
Three separate results cost a withdrawn claim to interpreting a result too eagerly — twice from a headline written before a confound check had finished. A loop that interpreted results would reproduce that failure faster than anyone could catch it.
The write-up already names the real risk as “a large, internally consistent body of work that nobody outside has stress-tested.” Automating generation without automating scrutiny makes that worse, not better. So every report carries a section headed what this page did not check — as a section, not a footnote.

What it cannot catch

Its first end-to-end run passed everything: preflight six of six, all four verdict branches exercised by the dry-run, zero unsorted failures, the verdict firing exactly as pre-registered, nineteen minutes, no intervention.

It also measured two of its six criteria in ways that could not answer their own questions — and one of those was a confound the ticket had named before the run.

Gates catch broken code and broken process. A construct-validity error passes every one of them, because the code does exactly what it was asked and the question was wrong. The only thing that catches it is a person reading the numbers and asking whether the measurement could have come out any other way.

Write the gates anyway. Just do not mistake a green harness for a sound experiment.

Source in ai/edit-slice/tools/. The rules behind it are on Practice.