Five days of edit-slice: every claim, and what happened to it

This is the record, not the argument. The argument is the previous post; this is everything around it, including the parts that failed.

It is written to be as useful where a claim was wrong as where it held, because most of them were wrong.

Where it started, where it is

The project charter, written on day one:

~50 CounterFact edits, GPT2-medium, ROME. Two hand-built probe sets per edit: consequences (forward) and grounds (backward). Measure contradiction rate in each. Deliverable is a two-panel figure and a number.

Five days later: zero edits have been applied. No two-panel figure, no contradiction rate. The project is named edit-slice and has not yet edited anything.

What exists instead is a delivered tool, a published claim, and four models measured — which came out of building infrastructure for the thing that didn't happen. Four of the charter's premises turned out to be false, and each was falsified by measurement rather than argument.

The charter's premise

Measure the contradiction rate between an edit and its grounds.

An edit makes its own grounds jointly improbable, not contradictory — and that is why the choice of what to retract belongs to a person rather than a method.

Why it failed

Measured: grounds are evidential, not deductive. Every case had a satisfying world — bodies are repatriated, families emigrate. There is no contradiction to count.

Replaced by

Joint implausibility, graded, reported with the facts carrying it.

Evidence

E-002, five hand-read cases

The claims ledger

Eight claims were tried and revised. Each row is a thing believed on a Monday and given up by a Thursday.

Every claim and its fate. Filled marks are claims that held; open marks were withdrawn or narrowed. The vertical axis is only ordering, not magnitude.

Seven withdrawn or narrowed, one standing — and the standing one was a by-product of infrastructure built for something else.

One claim's full life

Claim 1 is the best illustration of the whole pattern, because it went through every state a claim can occupy. Following it end to end is more useful than the ledger above.

Day 1 — confident. Backward probing is unexplored. Everything in the literature probes forward: edit a fact, check its consequences. Nobody probes the grounds that made the original fact hold.

Day 2 — suspicious. The claim was too comfortable, which is usually a bad sign. The obvious place for it to break was RippleEdits (Cohen et al., 2024), whose Logical Generalization criterion covers symmetric and inverse relations. Symmetric relations sound backward. If that is already a grounds probe, the claim isn't narrow — it's taken.

Reading the definitions rather than a summary turned up two things.

A bookkeeping one first: a note in another repo described that paper as EMNLP 2023. It is TACL 2024. Secondhand notes decay silently, and I had been treating mine as though they could not.

Then the definition itself. Logical Generalization says relations satisfy logical constraints — Sibling is symmetric, so if (e, Sibling, o) holds then (o, Sibling, e) does too — and checks whether those survive the edit. So it does probe something that looks backward: edit (Eiffel, located-in, Rome) and it asks what Rome contains.

But that is the same triple with its arguments swapped — the edit restated, not the reasons for it. The transitive case is sharper still: to get from "Eiffel is in Rome" to "Eiffel is in Italy" you must use the fact that Rome is in Italy, and the benchmark only ever checks the conclusion. Whether the premise survived is never asked.

That distinction became load-bearing enough to write into the project's binding definitions:

	what it means	status
Argument order	swap the arguments of the edited fact	covered by RippleEdits since 2024
Justification order	the distinct facts whose truth was a premise for the edited fact	not covered by anything we can find

The distinction the narrowing produced. Everything downstream depends on it.

"The Eiffel Tower was built for the 1889 Paris Exposition" is not the location fact inverted. It is a different fact about a different thing, and it is the one that goes stale in silence.

Day 2, continued — narrowed. Not backward probing is unexplored, but: existing ripple evaluation probes the logical closure of the edited triple; inverse and symmetric probes exist and are backward in argument order; nothing probes backward in justification order. Less exciting, more defensible, and it cost an afternoon. Better to find the counterexample to your own claim than have a reviewer find it in a room, in front of people.

Day 3 — withdrawn entirely. A second literature pass found CounterFact apparently checking possession already, so the broader framing was given up and a weaker one written in its place.

Day 5 — restored, and verified. Reading the ROME paper's construction appendix (Meng et al., 2022) rather than its prose showed the withdrawal was the error. Records are built "solely using the PARAREL entry", with no model consulted at any point.

Claim 1, after four revisions

Backward probing is unexplored — nobody probes the grounds that made a fact hold.

Existing evaluation probes the logical closure of the edited triple. Inverse and symmetric probes exist and are backward in argument order. Nothing probes backward in justification order — and CounterFact performs no possession check at all.

Narrowed by

Reading RippleEdits' definitions directly (day 2)

Withdrawn

Day 3, on a literature pass that misread the benchmark

Restored

Day 5, from the primary source's construction appendix

Status

Holds, verified

Three things that generalise, none of them profound, all of them relearned the hard way:

1. Read the primary source before designing around a summary. Every error in this sequence — the wrong venue, the almost-scooped claim, the false withdrawal — came from trusting a note or a search result over the paper.
2. Go looking for the strongest counterexample to your own idea, early. If it holds you have lost a week. If it doesn't you have gained a defensible sentence.
3. A claim getting smaller is progress — and a claim getting smaller wrongly is recoverable, provided the reason for each move was written down.

What we actually measured

Possession does not saturate

The one result that held. Possession means the true answer ranks first among 50 type-matched candidates and ranks higher with the real subject present than with it replaced by a placeholder — so a model scoring off the template's base rate earns nothing.

Possession against model scale, 165 rigid-relation items per model, 50 candidates, both arms. The dashed line is what the template alone achieves — the floor any real result must clear.

Monotonic, decelerating, still climbing at 405B. Fifteen percent of a curated factual benchmark is not held by the largest model reachable.

Scale buys the hard relations and leaves the easy ones flat. Native language sits at 80% on a 6B model and 93% above it. The movement is all in the strugglers — original broadcaster 13% → 87%, place of birth 13% → 53%.

"Does the model know it" has three different answers

Three defensible measures of the same 165 items on the same model. They are not in error; they measure different properties, and any single possession number is a choice reported as a fact.

The 12% is the one that matters for method. It is not ignorance: CounterFact's templates are ambiguous between temporal and locative readings — "Karolos Koun died at" invites " the age of 90" — so a model that knows the answer answers a different question. Under constraint the same model ranks Koun/Athens 1st of 10.

Routes taken and abandoned

Six routes were opened and closed. Each is recorded with what killed it, because a route that dies for a stated reason is reusable knowledge and one that just goes quiet is not.

1 · Mined logical rules over DBpedia — dead in 45 minutes

The plan: read probe questions off a knowledge graph instead of hand-writing them. A published benchmark had already mined Horn rules from DBpedia with AMIE, so the artifact existed. Read the rules backwards and the body of a rule whose head is the edited fact is its justification set.

What the rules actually contained:

?a birthPlace ?h   ?b isPrimaryTopicOf ?h   =>   ?a birthPlace ?b
?a hometown   ?g   ?g commonName       ?b   =>   ?a hometown   ?b

Read it: if a was born in ?h, and ?b is the same entity under another name, then a was born in ?b. That is preservation of a fact under renaming the object — a tautology about DBpedia's redundant naming predicates.

Of 12,675 mined rules, median support was 1. Of the 913 that survived a support/confidence filter, 66.5% were alias tautologies, 21.2% were headed by an alias predicate, and 2.3% were substantive — and those headed on gdpPppYear and topLevelDomain.

All 32 rules with a head matching one of our target relations were alias tautologies. Not thin: zero.

The deeper reason only became clear four days later. Rule mining finds chain patterns, and what we were asking for are star relations — see below. No amount of KG quality would have fixed it; we were asking a chain-finder for a structure it does not represent.

2 · AGM kernel contraction — dropped after five cases

The best idea of the week, and it lasted a day. Reframing forward/backward as expansion/contraction imports fifty years of belief-revision results instead of restating an observation: editors only ever add, contraction is where non-determinism lives, and Hansson's kernels give a formal definition of the justification set.

Then we read five actual cases:

post-edit state	contradictory?
died in Barcelona, buried in Baltimore	No — bodies are repatriated
developer IBM, publisher Sega	No — these differ routinely
named after Hamburg, located in London	No — things are named after distant places
native language Russian, born in Paris	No — families emigrate

None of them. Kernels are minimal entailing subsets; if nothing entails, there is nothing to break and contraction is undefined.

What survived was AGM's other half — the ordering machinery (entrenchment, Grove spheres, Lewis closest-world) rather than the closure machinery. We kept the account of the problem and dropped the algorithm.

And the loss strengthened the thesis. If grounds entailed conclusions, a machine could compute what to retract and human discretion would be a convenience. Because they don't, someone has to choose — which is the project's actual argument.

3 · The containment chain family — dead in a day, and our own table said so

To get a real contradiction we needed grounds that genuinely entail. Transitive containment looked right:

X located-in Y   &   Y located-in Z   ⊨   X located-in Z

Edit the outer to some other country and the two inners still entail the original. 62 chains mined from 120 seeds.

Then: probes/relation_modality.md, our own artifact from three days earlier, labels P131 mutable — "boundaries are redrawn." So editing the outer admits a temporal escape ("it is in Italy now") — the exact reconciliation that made the kernel cases non-contradictory.

We had written the table for precisely this purpose and did not consult it.

4 · The diagnostic framing — withdrawn, then restored

Claimed: editing benchmarks measure propagation into knowledge they never verified was there. A literature pass found CounterFact appearing to handle it, so the claim was withdrawn and a weaker one written up.

Reading the paper's construction appendix rather than its prose reversed that: records are built "solely using the PARAREL entry", with the counterfactual drawn by weighted sampling over the same predicate and no model consulted at any point. The benchmark's own table then confirms it — unedited GPT-2-XL scores ES = 22.2, meaning 22% of records already favour the false target.

Two literature passes missed this because both searched for competing work instead of re-reading the primary source. The paper was in a local directory the whole time.

5 · Pairing possession with a structure map — split, not merged

The tempting move: ship possession and a map of the belief neighbourhood around a fact. Rejected. Possession was complete and sharp; the structure map was neither, and its value was as foundation for an arc that was blocked. Pairing a finished thing with an unfinished one makes both worse.

6 · Grounds by directed intervention — right method, wrong order

Define g is a ground of f iff intervening on g moves f more than intervening on f moves g — causal and directed, discovered rather than asserted, which escapes the circularity of hand-written probes.

Still the right method. It was overtaken because a ground the model does not hold cannot be tested for direction, so possession had to come first. Parked, resumable.

Four structural findings

These came from sitting with the premise rather than from running anything. Each reorganised the project more than any measurement did.

The graph is a star, not a chain

For four days we said "the graph gives us grounds." It does not. Look at what our grounds had actually been:

Perec's native language is French. Grounds found: born in Paris · citizen of France · speaks French.

Every one is another property of the same subject. Wikidata asserts no edge between them. They are siblings on one node, and the justificational relation among them is entirely our imposition.

Three distinct neighbourhood structures that were being conflated. Grounds live in the star, where the graph asserts no edges at all; the existing literature traverses the chain.

Consequences, all recorded: rule mining failed because it finds chain patterns; and k-hop depth — which the whole design was built around — does not apply to the star at all, because a star has no path structure. The question "how far does it propagate backward?" was malformed.

Editors expand; they never contract

Forward propagation is expansion; backward is contraction. Every editing method — ROME, MEMIT, fine-tuning — is an expansion operator: it pushes probability toward a target. None has a contraction primitive.

Independent mechanistic support arrived from a paper we found while checking something else: a compact mask over ROME's edited weights restores the pre-edit answer in over 70% of cases. If the edit had overwritten anything, there would be nothing left to restore. Edits suppress.

Which makes the orphan worse than modelled: the model ends up holding the new answer, the old answer, and the grounds for the old answer, simultaneously.

In-degree and out-degree are different variables

Out-degree — how many consequences a fact has — is the forward propagation burden, and it is already studied. In-degree — how many independent justifications — is the contraction burden, and it is not. Median in-degree in our data is 8 co-predicates per subject; 1% of subjects have none; 84% have three or more.

The decision nobody makes

The thesis, arrived at sideways. If logic cannot say which belief to give up, then an editor that silently picks one is making an unaccountable epistemic choice — at scale, invisibly, with nobody able to inspect or contest it.

So the orphan is not merely a technical defect. It is a decision that got made without anyone making it. And an instrument's job is to surface the choice, never to make it: show the structure, never the ranking.

Attrition is where the artifacts live

Every number here is a rate over a surviving subset, and the survivors are never a random sample. Three funnels, each of which produced or nearly produced a false result.

Three selection funnels. In each, the kept fraction is biased toward whatever made survival easy — which is exactly where a headline number goes wrong.

The middle one is the instructive failure. The possession filter's first end-to-end run reported 92% held — and had scored 12 of 40 items. Candidates were being drawn from the edit set itself, so relations with fewer than three attested answers could not form a candidate set and were dropped silently. What survived were the two relations dense enough to be easy.

A forty-point inflation produced by a silent denominator change — which is precisely the selection artifact the tool exists to detect, sitting inside the detector. It was caught by running it, not by reading it.

The fix is now part of the tool: candidates come from a reference vocabulary separate from the edits under test, coverage is always reported, and a warning fires below 80%. A held-rate over a silently biased subset is worse than no held-rate.

Where the chains stand now

The family was re-anchored this morning on a relation that cannot be relocated.

The current entailment family. The outer fact is rigid — birth cannot be relocated — which closes the temporal escape that killed the containment version.

Editing the conclusion leaves both premises standing, and they still entail the original. Birth cannot be relocated, so the escape that killed the containment version is closed: the model must give up "born in Paris" or claim "Paris was in Germany" — both specific, both directly probeable.

136 chains from 200 seeds. P17 also fixed a type bug by construction: walking the administrative hierarchy had produced "Gracie Mansion is located in the country of" → "New York City", the same error class we had published about two days earlier. P17 is defined to take a country, so the template's type now holds without a filter.

Gating them: the same chains, two scales

A chain is only usable if the model already holds all three of its facts. A model that never knew "Vlaminck was born in Paris" cannot be caught failing to give it up, so the gate runs before the edit, not after. Each fact is scored against a pool of type-matched candidates and must both rank first and rank higher with the real subject than with a placeholder.

Possession rate per chain position across three scales, over the same 136 chains and the same candidate pools. One hue in three steps, because model scale is ordered. The bottom group is the conjunction: all three legs held.

74 usable chains at 70B, 58 at 8B, 20 at GPT-J — 54%, 43%, 15%. The edit experiment was blocked on having enough chains a model demonstrably holds end to end. It is not blocked now.

The 8B row is the one that mattered, and it was only run because the question was asked backwards. The plan was to solve editing at 70B: ROME there needs second-moment statistics over d_mlp 28672, roughly 3.3 GB, which had been neither collected nor costed. That obstacle existed entirely because we had assumed 8B would not clear the possession bar — at the one scale where editability was already verified. Nobody had checked. It clears at 43%.

And once E-011's scope limit is applied — the measure is undefined where the answer is the modal one for its relation — the picture sharpens considerably:

model	inner_1	inner_2	outer	usable
GPT-J-6B	45%	92%	37%	20/78 = 26%
Llama-3.1-8B	73%	97%	77%	51/78 = 65%
Llama-3.1-70B	87%	99%	85%	64/78 = 82%

On the domain where the instrument actually works, 8B holds two thirds of the chains end to end and outer stops being the broken leg — 49% becomes 77%. Fifty-one usable chains is more than enough for an existence claim, which is all this pilot licenses.

So the expensive problem gets dissolved rather than solved. 70B still offers a quarter more chains, and it would cost an uncosted covariance collection plus an editability result we do not have. The edit runs at 8B.

The more informative half is what did not move. outer is the worst leg at both scales, and by a similar margin. The tempting reading of the GPT-J result was that a 6B model is simply too thin to hold a birth country; that reading is now dead, because a model eleven times larger has the same shape of deficit. What survives is the candidate pool. The country pool has 28 distinct fillers, with the United States alone accounting for 40 of 136 and the top four for 57%; the city pool has 78 distinct fillers, and its top four reach 16%. A concentrated pool makes first-place ranking harder for reasons that have nothing to do with whether the model knows the answer.

The first explanation, and it was wrong within the hour

The outer deficit comes from candidate-pool concentration — countries have 28 distinct fillers against 78 cities, so first-place ranking is harder.

"X was born in the country of ___" completed with " United States" is ungrammatical, where " France" is fine. The penalty lands on the two most common answers in the set, which is why it looked like a property of the model and is in fact a property of our prompt.

Why it failed

The deficit is not spread across the pool at all. It is two strings. United States 6/40 held, United Kingdom 2/14 — while France is 13/13, India 5/5, Japan 4/4, Spain 5/5. Grouped: 15% for article-taking names against 84% for bare ones.

Why it looked right

The countries with definite articles are also the most frequent ones, so concentration and surface form were perfectly confounded in the numbers I looked at first.

Also tested and killed

That the placeholder control was misfiring on prior-favoured answers. Exactly 1 of 136 items fails that way.

Replaced by

A template artifact: we score a Wikidata label, not a natural continuation.

Evidence

Re-analysis of data already on disk, no new compute

28 chains are blocked only by a US/UK outer leg. If the artifact reading holds, usable goes from 74/136 to 102/136 — 54% to 75% — and the finding that survives is about our instrument rather than about Llama.

The honest summary of the day is that this is the third member of one family, after the denominator bug and the template check. Every one was a selection effect living inside a measurement tool; every one was found by looking harder at data already on disk, not by collecting more. That ratio is worth stating plainly, because it is the opposite of what a compute budget assumes.

The legs are not independent, and that is the useful number

The decision to move to 70B was written down the day before with a prediction attached: "possession is 56% on GPT-J against 79% at 70B, so the usable set should roughly double." It went up 3.7×. The estimate treated a per-fact rate as if it carried straight over to a per-chain one, and a chain is a conjunction — its rate moves faster than any leg in it.

Having been wrong that way, the honest next question is what the conjunction actually does. Multiplying the three marginals predicts 4.5% usable at GPT-J. The measured figure is 14.7% — 3.27× what independence gives. At 70B the same comparison is 43.6% predicted against 54.4% measured, a lift of only 1.25.

pair	GPT-J lift over independence	70B lift
inner_1 & outer — share the subject X	2.10	1.15
inner_2 & outer	1.53	1.09
inner_1 & inner_2 — share no subject	1.26	1.00

The lift is ordered by shared subject and flattens with scale. The obvious story is that possession clusters per entity — a chain about a well-known person holds at every leg, one about an obscure person at none — so the binding variable would be subject familiarity rather than leg difficulty.

That story is a claim about a variable nobody had measured, so I measured it. Prominence, taken as the subject's statement count in the Wikidata snapshot already on disk, covers 136 of 136 chains and spans 5 to 404. Stratifying by it leaves the lift completely untouched: 2.08 within-stratum against 2.10 pooled at GPT-J, 1.15 against 1.15 at 70B. A common cause would have collapsed the within-stratum figure toward 1. It did not move at all.

Prominence is a real variable — it predicts the possession level at 70B, where inner_1 runs 78% to 93% and outer 46% to 68% from the low to the high tercile. It is simply not the one binding the legs. And at 70B the lift is ~1.1 whichever way the set is cut, which is close enough to independence that my framing of it as a finding was overstated. What remains unexplained is a residual 1.61 at GPT-J.

This is the number worth carrying forward, because chain attrition is the cost model for every longer chain anyone might build. Under independence a five-hop chain at 70B would be hopeless. Under the dependence we measured it is merely expensive. Costing a multi-hop design off marginals alone will under-build it — which is the same mistake as the prediction at the top of this section, one level up.

The test that came back Deny, and was more useful for it

The surface-form hypothesis was worth one run, so it got a ticket with its outcomes written down in advance: confirm if rendering countries naturally recovered the article-taking names, deny if it did not. The same 136 items, the same model, the same 28-country pool, one thing changed — how every candidate is spelled.

Held rate went from 19% to 16%. On the headline number, Deny.

The rank data underneath says the opposite, and the pair is the actual result.

condition	answer class	n	ranks 1st with subject	ranks 1st with placeholder	held
bare	article-taking	58	21%	2%	19%
bare	bare-name	78	85%	0%	85%
natural	article-taking	58	69%	69%	16%
natural	bare-name	78	83%	0%	83%

Surface form was real: spelling it " the United States" moves rank-1 from 21% to 69%. The model was never failing to know where these people were born. It was failing to complete an ungrammatical string.

And fixing that destroys the control. Under the natural spelling the placeholder prompt ranks the same answer first at exactly the same rate — 69%. Of the 40 items that now rank first with the real subject, 31 are rejected by the lift test, every one of them because the placeholder ranked it first too. "[X] was born in the country of the United States" is the modal completion whether or not [X] means anything at all.

So the two criteria have no common operating point. A spelling that makes the answer rankable makes it prior-favoured; a spelling that suppresses the prior does it by being ungrammatical. There is no spelling that satisfies both — which means our possession filter cannot certify possession of a fact whose answer is simply the most common answer for its relation. Bare-name countries sit at 0% placeholder rank-1 under both spellings, so the measure is sound there and undefined here.

That is a limit on the tool this project ships, not a fact about Llama, and it is the most useful thing found all day. It also retracts the 102/136 ceiling from two sections ago: under the natural spelling, usable chains go 74 → 71.

The fix is a better control rather than a better spelling. Asking "does the subject matter at all" is defeated by a modal answer by construction. The question worth asking is "does the model track which subject" — score the answer against a paired real subject whose true answer differs, and require the model to prefer the right one. That discriminates knowledge from prior without needing the prior to be low. It is opened as E-012, and deliberately not retrofitted into any number already published here.

Both fixes, and an interaction nobody predicted

E-012 built that control: for each item, take the score of its true answer under every foil prompt — a real subject whose answer differs — and ask what fraction it beats. That is an AUROC. Chance is 0.5. It is indifferent to how high the answer scores overall, which is exactly why it is defined where the placeholder test is not.

Run alone, it gained one chain. Modal answers discriminated well — median AUC 0.901 — but the row test now blocked them, because " United States" still would not rank first while it was still spelled ungrammatically. Thirty-two items sat on that intersection: control satisfied, ranking not.

So the last run was the full 2×2.

Usable chains out of 136, Llama-3.1-8B, over rendering × control. The lines are parallel until both fixes are present; the gain is an interaction, not a sum.

Neither fix alone does anything: +1, and +1. Together, +20.

That is an interaction, and it explains the whole shape of the day. Each fix removes a different blocker, and an item needs both removed before it passes. Rendering fixes given this subject, which answer; the paired control fixes given this answer, which subject — and the control fix is required because the rendering fix is what makes the answer prior-favoured in the first place. Test either in isolation and it reads as a null result. We did, twice, and it did.

On outer, the leg that had been broken all day, modal answers go from 25% to 68% on the row test and 25% to 64% held.

And the comparison that actually stings: Llama-3.1-70B under the old instrument gave 74 usable chains. Llama-3.1-8B under the fixed one gives 78. Repairing the measurement was worth more than an order of magnitude of scale — on the same 136 chains, with the smaller model.

The obvious objection is that the new control is simply more permissive. The 2×2 answers it: on bare rendering it gives 59 against the placeholder's 58. It is not looser. It is defined in a region where the old one was not, and that region is where every modal answer lives.

Two threats stay on the record. Foil prompts differ from the item in subject frequency as well as identity, and prominence predicts possession level, so the AUROC is not corrected for it. And inner_1's foil coverage is 63%, because its 78-city pool is larger than the candidate budget — its AUC rests on a sampled foil set, where outer and inner_2 are dense at 100%.

Infrastructure, which is most of the work and none of the interest

what	why it exists
Batched NDIF scoring, 8.3× faster	Compute on a 70B model is sub-second; the ~10s is queue and transfer. Round trips are the budget, not FLOPs — the first design had that backwards and I quoted four hours based on it.
Adaptive OOM halving	Per-process memory budgets differ across deployments (13.5 GB on one, 17.7 GB on another). Any fixed batch constant is wrong somewhere.
logit[t] − logsumexp instead of log_softmax	The latter allocates a second [batch, tokens, 128k-vocab] tensor. That, not batch size, was the actual OOM — and it was 2.5× worse on Llama than GPT-J purely from vocab size.
Transport errors classified by module, not type name	httpx.ConnectTimeout matched none of my hardcoded names and killed a 165-item run at item 28.
Dated, checksummed snapshots	Wikidata mutates. A result is not reproducible without the bytes that produced it. Snapshots are additive — one directory per ingestion, never overwritten.
Per-item caching everywhere	Three separate debugging cycles cost minutes instead of restarts. This is worth more than any of the fixes above.

What the five days actually consisted of.

The pattern in the errors

Eight reversals in five days. Sorting them by where the error sat produces the only generalisation in this post worth carrying:

None of the errors were in the measurements. Every number held. What failed each time was the framing supplied around the number — the mechanism I invented for a phenomenon that had been measured correctly.

• The possession curve held; the story about why CounterFact's filter was weak was wrong twice, in opposite directions.
• Grounds were correctly found to be evidential; the claim that this made containment a contradiction was wrong.
• The alias tautologies were correctly counted; the theory that better data would fix them was wrong.

And a second, sharper pattern: each error was already refuted by something we had written down.

the error	what already said otherwise
Containment gives a strict contradiction	relation_modality.md labels P131 mutable — written three days earlier, for this purpose
Outer template says "country of"	T-044, the template-ambiguity finding we had just published about
"The graph gives grounds"	definitions.md declaration 4 — the graph is a normative audit spec, not a model of the network, written day one
CounterFact verifies possession	The paper's own appendix, three pages from the sentence misread
Grounds are better known than heads	The grain confound, visible in the same table

Every wrong turn, and the artifact that already contained its refutation.

The apparatus was not failing to record things. It was recording them accurately and going unread. That is a different problem from disorganisation, and a more uncomfortable one.

The practical rule extracted from this: when a claim turns on what a paper or artifact did, read its methods section before concluding from its prose. Aspiration and mechanism live in different places and only one of them is binding.

Six analogies, and where each one breaks

These are for explaining, not for claiming. An analogy that isn't bounded gets used past its evidence, so each comes with the place it stops working.

A drug trial that never checks the patients were sick

You give a cholesterol drug to a hundred people and measure whether cholesterol drops. It doesn't drop in forty of them. But forty of them had normal cholesterol to begin with, and nobody checked. "The drug failed in 40%" is now a statement about recruitment, not about the drug.

The edit is the drug, the fact is the condition, and CounterFact never screened for the condition — records are built from a knowledge graph with no model consulted at any point.

Where it breaks: a drug trial has a control arm. Editing evaluation does not, and cannot easily get one — there is no "unedited but otherwise identical" model to compare against except the model itself, which is what the prior-arm control in the possession measure is standing in for.

"Does this student know the material?"

exam	score
Multiple choice, one obviously-wrong distractor	75%
Multiple choice, fifty plausible options	61%
Open-ended essay, ambiguously worded question	12%

Same student, same knowledge, three defensible exams. None is cheating. The third is not ignorance — it is the student answering a different question than the examiner meant, which is exactly what "Karolos Koun died at" → " the age of 90" is.

So any single possession number is a choice about the exam, reported as a fact about the student.

Circumstantial evidence, not a proof

"He was buried in Baltimore" is evidence that he died there. It is not proof; bodies get shipped. So when you edit "died in Barcelona", the burial record does not become false — it becomes odd. Nothing is contradicted. The story simply stops hanging together.

Against that: "born in Paris" and "Paris is in France" entails "born in France". That is a proof. Change the conclusion and something has to give.

Almost everything surrounding a fact is circumstantial, which is why the chains had to be hand-built — to manufacture the rare case where the evidence really is proof.

The graph is a filing cabinet, not a citation trail

Someone's birth certificate, passport and employment record all sit in the same folder. The folder records that they concern the same person. It does not record that the passport depended on the birth certificate. That dependency lives in the clerk's head, not in the filing system.

We spent four days saying "the graph gives us grounds". It gives us the folder. The dependencies were always ours.

Where it breaks: real archives sometimes do record provenance — that is what a chain of custody is. Wikidata mostly does not, and the properties that look like provenance (described by source, different from) are bookkeeping rather than justification.

A redirect, not a deletion

The file is still on the server; a forwarding rule was added in front of it. Remove the rule and the original serves again.

That one is close to literal. A compact mask over ROME's edited weights restores the pre-edit answer in over 70% of cases — if anything had been overwritten there would be nothing left to restore.

So after an edit the model holds the new answer, the old answer, and the grounds for the old answer, simultaneously.

The 92% that was really 52%

Armour the bombers where the returning planes were hit, and you armour exactly the wrong places — the planes hit elsewhere never came back to be measured.

The possession filter's first run reported 92% and had scored 12 of 40 edits. The 28 that dropped out were the hard ones. The same survivorship error, inside the tool built to detect it.

And the one that covers all of them: constellations

The stars are really there. Position, brightness, distance — measurable, checkable, and everyone who looks sees the same ones.

Orion is not there. The lines are drawn by us, and different cultures drew different figures from identical stars.

That is the finding. A model's facts are real and measurable — 56%, 74%, 79%, 85%, reproducible across runs. But every relation we tried to measure between them — grounds, dependency, entailment, justification — turned out to be a line we were drawing rather than a line we were finding. We kept pointing the telescope harder at the gaps between the stars.

The useful part is that this does not make constellations worthless. Sailors navigated by Orion for millennia. It means only that you have to say "here is the figure I am using, and here is why" rather than "here is the figure that is up there."

Which is the whole argument for human discretion. If the lines were really in the sky, a machine could trace them. Because we draw them, someone has to choose — and that someone should be a person who can be asked to justify the choice.

Where the evidence points

Three statements, in increasing order of how much they cost to learn.

The knowledge is thinner than everyone assumes. Models hold 56–85% of a curated benchmark's facts, not ~100%, and nothing in the standard pipeline checks.

"Knowing" is not one property. Three defensible measures gave 75%, 61%, and 12% on identical items. That is not measurement error — they measure different things and all three are legitimate. Any single possession number is a choice being reported as a fact.

And the structure we kept looking for was not there. We wanted a dependency graph: the KG asserts no edge between a fact and its grounds. We wanted entailment: support is evidential and every "contradiction" had a satisfying world. We wanted kernels: no minimal entailing sets exist. We wanted a decidable oracle: the mined rules were naming tautologies.

Where five days of evidence points

The facts are real. The structure between them is ours.

Status

Synthesis, not a measured result — a reading of the measured results

Consequence

An audit standard must be argued for, not discovered

Tension

This was written down as declaration 4 on day one and forgotten repeatedly

A model's dispositions are genuinely there and genuinely measurable. But every relation we tried to measure between those dispositions — grounds, dependency, entailment, justification — turned out to be something we supplied rather than something we found.

That is not deflating, because it changes the kind of claim available. Not "here is the dependency graph inside GPT-J" but "here is the audit standard I am holding it to, and here is why." The first is a discovery that keeps failing to replicate. The second is a position that can be argued with.

And it is why the discretion argument holds. If grounds entailed conclusions, a machine could compute what to retract. Because they don't, someone has to decide — which is the project's thesis and its longest-running embarrassment, turning out to be the same observation.

The edit finally ran, and it said no

Everything above was a gate on one experiment. It has now run.

Seventy-one chains survived the cleaning, forty-two survived a week of NDIF outages, and each got two rank-one edits at layer 5 — ROME as configured by EasyEdit, which is to say with mom2_adjustment: false, the unwhitened update every one of its fourteen shipped configs uses.

Edit the conclusion — "Vlaminck was born in the country of" → Germany — and then ask where he was born. The result is unambiguous and, for a while, thrilling:

inner_1 lands in the target country	rate
baseline	5%
after the real edit	67%
after a same-subject control edit (his occupation)	5%
placebo country (the null)	0%

Edinburgh becomes Hamburg. Paris becomes Santiago. Tokyo becomes London. The model does not merely lose the old answer — the control loses it too — it moves the answer somewhere consistent with the new conclusion, and the control does not. Thirteen times the control rate, placebo at the floor.

It reads exactly like the thing this project was built to find: the model gives up the premise it can afford to give up — which city — and keeps the one it cannot — that city is in that country. Defeasible yields, necessary survives. Belief revision, measured.

The control that killed it

That reading has a cheaper rival, and the rival predicts the same data. The edit makes inputs keyed on the subject emit a Germany-valued vector. The birth-city prompt shares the subject, so its key is similar, so it receives a dose of the same vector. A Germany-flavoured residual stream promotes Germany-associated tokens, and German cities are Germany-associated. No inference anywhere — just content bleeding through a similar key.

The occupation control cannot separate these. It varies content, so it shows content matters; it cannot show whether the content licenses an inference.

The control that can holds subject and target country fixed and varies only the relation:

edit	licenses "born in a German city"?
X was born in the country of → Germany	yes, deductively
X works in the country of → Germany	no

Both push the same country-valued vector through the same subject's key. Only one licenses the implication. Nobody infers a birthplace from an employer's country.

arm	lands in target country
birth-country edit	28/42 = 67%
work-country edit	28/42 = 67%
paired difference	+0.0 pp

Exact McNemar on the discordant pairs: p = 1.000, nine each way. Capital share among hits identical at 82%. Where both arms land in the target country, they choose the same city 84% of the time.

Editing where a person works relocates their birthplace exactly as often as editing where they were born.

The result the project was built to find

Editing a conclusion makes the model revise the defeasible premise while retaining the necessary one — belief revision, measured.

The edit produces displacement, not revision. What looked like the model choosing which premise to sacrifice was the same vector being read twice.

Why it failed

A work-country edit relocates the birthplace at an identical rate (67% vs 67%, paired gap +0.0 pp, McNemar p = 1.000). Work country licenses nothing about birth city, so the relocation cannot be an inference.

What it actually shows

Rank-one editing displaces probability toward country-flavoured content in any probe sharing the subject. Country-specific, systematic, and indifferent to whether the relation licenses the implication.

Caught by

A dispute filed three days before the number existed, after the adversary agent surfaced our own binding definition against a claim about to be published.

Evidence

E-015, 42 chains, subject and target held fixed

The city that ate a third of the results

One number in that run is worth more than the headline. Across the birth arm, Washington, D.C. absorbs 14 of 42 destinations — 33%. In the work arm, 11 of 42.

It is the highest-prior city in our candidate pool, and it is where probability goes when no coherent relocation happens. Which makes it the third appearance of one pattern: the US being 40 of 136 countries, article-taking names losing to bare ones, and now a single city acting as a sink. Concentrated pools with one dominant attractor have shaped three separate results in this project, each time arriving disguised as a finding about the model.

Any pool we report from here reports its attractor mass alongside it, the way out-degree has to report its edge vocabulary.

What survives

The measurement stands; only its meaning shrank. Rank-one editing at this configuration produces country-specific displacement into same-subject probes at 67% against a 5% control — that is real, and it is a fact about what these editors do to knowledge they were not aimed at, which is the charter of this project.

What does not survive is the inference. The model is not reasoning about what its new belief implies. And the honest version of the orphan story gets worse rather than better: the edit does not retract the grounds, and it does not coherently rewrite them either. It smears country-flavoured probability over anything that mentions the subject.

Whitening could not have changed the answer, and the reason is one line of algebra

The last post left a gap open and called it "the obvious place a genuine inference effect could still hide": everything had been run with C = I, the unwhitened update EasyEdit ships for all fourteen of its ROME configs.

It is not hiding there. It cannot be.

Start with what ROME's update actually does to an input it was not aimed at. The edit is built from three things: the key read at the subject's last token of the edit prompt, the edit vector the layer will add to its output there, and a direction along which the weights are moved. Only two of them matter here — call the key k* and the direction u. The whitening choice is entirely a choice of u: EasyEdit's default is u = k*, and the whitened version is u = C⁻¹k*, where C is the second moment of the layer's keys.

For any other input with key k, the edited layer's output shifts by the edit vector scaled by a single number:

coefficient = (k · u) / (u · k*)

That is the whole mechanism of leakage. The edit reaches an unrelated input exactly as far as that coefficient is large.

Now take a probe that shares the edit prompt's prefix up to the subject's last token — "Jack Marshall was born in the city of" against the edited "…in the country of". Under causal attention the key at that position cannot see the later tokens, so it is the same key: k = k*. Substitute:

coefficient = (k* · u) / (u · k*) = 1

for any u. The normalisation divides by exactly the quantity it is about to produce.

The gap named in the last post

Everything so far uses C = I, and a whitened editor is the obvious place a genuine inference effect could still hide.

Whitening can only reduce leakage to prompts that do not share the subject's prefix. Against the probes this project is built on, it is a no-op by construction.

Why it failed

ROME normalises its coefficient by u·k*. At any position where the probe's key equals k*, the coefficient is 1 for every choice of u — so no covariance can attenuate it.

Confirmed by

42/42 chains, identical top-1 destination to C = I at all three λ (7.2e-02, 7.2e-03, 7.2e-04).

What it cost to close

One coefficient profile along the probe prompt. The argument was available before the experiment and nobody looked.

Evidence

E-016, whitened editor via Woodbury, N=2048 keys at layer 5

The measurement agrees, and agrees boringly: forty-two chains, three values of λ, and the birth arm's top-1 destination is identical to the unwhitened run in 42 of 42 every time. The profile along one probe shows why — coefficient 1.0000 at the subject's last token under both editors, and only after that do the two diverge (0.90× at the next token, 0.28× by the preposition).

The cosine said it was a no-op, and the cosine was wrong

The gate that authorised this experiment nearly closed it with the right answer for the wrong reason. Whitening barely rotates the update: cos(k*, C⁻¹k*) = 0.97. Read at speed, that says the whitened editor is the same editor and there is nothing to test.

It is not the same editor. On held-out keys the mean absolute coefficient drops from 0.0260 to 0.0013 — a twentyfold narrowing of reach — at a cosine of 0.97. A near-identical direction with a radically different blast radius is not a contradiction; the coefficient depends on where u points relative to the rest of the key distribution, which is precisely what a 0.97 cosine does not measure.

Had the gate stopped at the cosine, [E-016] would have closed with "whitening does nothing" — true of the outcome, false about the editor, and useless as a reason. That is the second time in this project the aggregate statistic and the component statistic disagreed and the component was right.

The escape hatch, tested and closed

The analytic account had one hole, and it was stated when the account was: it requires the probe to share the edit prompt's prefix, which our probes do by construction. A probe mentioning the subject later — "The city where Jack Marshall was born is" — shares no prefix, and its coefficient is not pinned.

Moving the subject later was predicted to break the pinning. It attenuates it by seven percent.

probe form	mean coefficient	median	min
early — the edit form	1.000	1.000	0.998
a different relation — died in the city of	1.000	1.000	0.998
late clause — the city where X was born is	0.935	0.954	0.670
possessive — the birthplace of X is	0.934	0.966	0.483
long preamble	0.925	0.946	0.707

The second row is the one that matters, and it is a case the analytic account did not construct. A probe asking about a different relation entirely scores exactly 1.000 against a birth edit's key, because the subject sits at the start and causal attention makes the key identical. The relation being asked about is irrelevant to how much of the edit arrives.

So the layer-5 key at a subject's last token is largely determined by the subject tokens themselves rather than by what surrounds them, and no natural reformulation escapes.

Same-subject leakage in ROME is structural — with a boundary I did not have when I first wrote that sentence. Two claims were bundled in it, and a layer sweep separated them.

The analytic one holds everywhere. A probe sharing the edit prompt's prefix — including one asking about an entirely different relation — scores exactly 1.000 at every layer tested, from 0 to 31, minimum equal to maximum. That is not an empirical regularity; it is (k*·u)/(u·k*), and it cannot come out otherwise.

The empirical one is an early-layer fact. Probes that reformulate rather than share the prefix — a late clause, a possessive, a long preamble — decay steadily with depth:

	layer 0	layer 5	layer 12	layer 24
late clause	0.986	0.935	0.795	0.524
possessive	0.990	0.934	0.794	0.564
long preamble	0.979	0.924	0.764	0.444

So "regardless of where the subject sits, or how much preamble precedes it" is true at layer 5 — the layer EasyEdit's config edits — and false by layer 24, where a reformulated probe receives roughly half the edit vector. The subject key is largely determined by the subject tokens early on, and stops being so with depth.

Layer 31 reads 0.699, which looks like a recovery and is not one. The different-subject floor there is 0.669 — everything scores about the same whoever is named. Carrying that floor at every layer is the only reason the difference between layer 24 and layer 31 is visible at all.

The decay is a change of direction, not of size

The coefficient is not a cosine. It factors:

c = (|k| / |k*|) · cos(k, k*)

which means a falling c has two possible readings — the key points somewhere else, or it points the same way and is smaller. Those are different claims, and separating them costs one extra number per measurement.

It is entirely direction. Across the reformulated probes, cosine falls 0.984 → 0.471 from layer 0 to layer 24 while |k|/|k*| never leaves 0.97–1.24. The ratio actually drifts upward, so magnitude works slightly against the decay: at layer 24 the coefficient reads 0.511 while the underlying alignment is 0.471, propped up by keys about 9% longer than k*.

Norm turns out to carry nothing at all here. Over eight layers, six probe forms and sixteen chains — including probes naming a completely different person — |k|/|k*| sits within roughly ten percent of one. Subject-position keys are all about the same length. Everything that distinguishes them is orientation.

That also says what layer 31 is. The different-subject rise to 0.669 is cosine (0.660, with the ratio flat at 1.009), so at the final layer the subject-position keys of different people are pointing two-thirds of the way toward each other. Not a scaling artifact, and better than the "noise" I had a paragraph ago: near the output these representations converge on a shared direction, and that is why the measure stops telling people apart.

What this does not show is why the direction moves. Attention progressively mixing context into the subject position is the obvious candidate and it is untested — a falling cosine is equally consistent with MLP writes at that position, or anything else that reorients the residual stream. "Cosine fell" is not "context was mixed in", and the ordering across probe forms at layer 24 does not obviously favour the mixing story anyway.

That sentence had a hole in it when I first published it, and it is worth saying how it got closed. Every control behind the 93–100% figure varies the form and holds the subject fixed. None of them varied the subject. So "any prompt containing the subject" was not actually separated from "any prompt" — the floor was assumed, never measured, and [E-016]'s own anisotropy number was a reason to doubt it: these keys sit on an effectively 26-dimensional manifold out of 2048, where arbitrary vectors have substantial overlap by construction.

Measured now. A different person in the identical template — "Marie Curie was born in the city of" scored against a Jack Marshall edit's key — comes back at 0.082, against 0.958 for the same-subject forms. The distributions do not touch: the highest different-subject coefficient is 0.153, the lowest same-subject one 0.483.

So the pinning is keyed on the subject, and now demonstrably so. The anisotropy worry was a good reason to run the control and a bad prediction of what it would say.

This also finishes the account of the result that closed the last post. The work-country edit and the birth-city probe both begin with the subject, so the probe receives the work edit's full, unattenuated delta. The +0.0 pp gap was not a null needing explanation — it was structurally required. No choice of C could have returned anything else.

Two mechanisms I guessed, and both were wrong

Before measuring the coefficient profile I proposed two accounts of why whitening changed nothing, and wrote both down. Neither survived.

The first was that the probe's key is aligned with k*, so whitening preserves it along with everything else. Measured: cos = 0.034. They are near-orthogonal, and the alignment story was not close.

The second was that the destination is scale-invariant — that the edit's magnitude does not matter, only its direction, so shrinking coefficients would not move the answer. Measured: a uniform ×0.27 rescale breaks relocation in three of four chains. Scale matters.

The third account was not a guess. It came out of the per-position profile, which was the one thing neither guess had looked at. An earlier "3.7× attenuation" figure I had been carrying turned out to be measured at the probe's final token, which contributes 0.03 of a 1.66 total — two percent of the mass. It described a negligible part of the update accurately.

Honest limits

The relocation half of the late-probe test is unrun, not negative. The late form holds the true city at baseline in 8 of 42 chains against 29 for the early form, which is below the floor of 10 set before the run. Reporting it as a weak result would have read as confirmation of the hypothesis under test, when the cause is that the probe does not work.

Some subjects do attenuate materially. The minima in that table — 0.483, 0.670, 0.707 — are real. This is an existence-and-tendency claim about prompt forms, never a rate.

Sixteen chains for the coefficient sweep, forty-two for the edit arm. One model, one layer, one relation family.

The different-subject floor was measured with one pairing. Subjects were paired by rotation. A pairing chosen to be adversarial — matched on token length, or nationality — could score higher, and was not run. The claim is that these different subjects score 0.082; that different subjects generally do is an inference from a clean separation, not a measurement of the space.

The sweep ran, and it narrowed the claim rather than confirming it. The form-robustness half is early-layer only; see the table above. What survives at all depths is the prefix-sharing case, which is algebra rather than measurement.

Delivery, never effect. Every number here is how much of the edit vector arrives, not whether it changes an answer. A ×0.27 rescale broke relocation in three of four chains, so a fall from 0.93 to 0.52 is likely to matter behaviourally — but that is an inference, and no edit was run at any layer but 5.

Eight layers is a shape, not a per-layer claim, and why the decay happens is unmeasured. That the subject key becomes context-sensitive with depth describes the measurement; it does not explain it.

Do not read this as a proposal. That reducing same-subject leakage requires changing the normalisation or the key selection rather than the covariance is a consequence of the finding, not a method I am offering. Any approach that limits an edit's blast radius by reweighting u inherits this limit.

Why the null was forced

A null result asks for an explanation, and the obvious suspect was the editor. Everything above used C = I — the unwhitened update, which drops roughly eight nats on facts about the subject that have nothing to do with the edit. A blunt instrument smears; maybe a sharper one would show the inference the blunt one destroyed.

ROME's method section is built on the sharper version: u = C⁻¹k*, where C is the second-moment matrix of the layer's keys. EasyEdit ships fourteen ROME configs and every one of them sets mom2_adjustment: false, including the two models the ROME paper itself used. So the whitening term the method is named for is off by default in the toolkit most people run.

Turning it on is not free — C is 14336² at this model, 0.82 GB, and inverting it is worse. But the key distribution turns out to live on a 26-dimensional manifold: half the variance in 16 directions, 99% in 338. So the honest model is low-rank plus a ridge, and Woodbury computes C⁻¹k* while forming nothing bigger than an N×N matrix. Seventeen megabytes instead of eight hundred.

The statistic that nearly ended the investigation

The first check was the cosine between k* and C⁻¹k*. It came back 0.97 — whitening barely rotates the update. On that number the whole line of inquiry closes: whitening changes nothing, move on.

It was the wrong statistic. ROME changes an arbitrary input by (v* − Wk*) · (u·k)/(u·k*), so what whitening does is shrink that coefficient on unrelated inputs — and a 0.97 cosine is perfectly compatible with a twentyfold change in it. Measured on held-out keys: 0.0260 unwhitened against 0.0013 whitened. The update's direction barely moves; its reach changes by a factor of twenty.

(The first version of that measurement evaluated on the same keys that defined C, which gave a spectacular and meaningless 1000×. Refit on 1400, evaluated on 665 held-out. The same denominator mistake as day two, in a new costume.)

So: a demonstrably twentyfold more selective editor. Run it on the same 42 chains.

Forty-two out of forty-two identical destinations. Paired gap +0.0 pp at every λ.

The coefficient is pinned, and it is pinned by arithmetic

Three guesses at why, all wrong, all cheap to have tested first. That the probe key was aligned with k* — it is nearly orthogonal, cosine 0.034. That the destination was scale-invariant — rescaling by 0.27 breaks relocation in three chains of four. Only the third attempt measured instead of predicting, and printed the coefficient at every position of the probe:

position	token	coeff C = I	coeff C⁻¹
7	'amin'	0.0918	0.0786
8	'ck'	1.0000	1.0000
9	' was'	0.1156	0.1040
14	' of'	0.0305	0.0086

Exactly one. Not approximately.

k* is read at the subject's last token of the edit prompt. A probe that begins with the same subject has an identical key at that position — causal attention guarantees it, since nothing differs in the tokens before. ROME divides by u·k*. So the coefficient there is (k*·u)/(u·k*) = 1 for any u, hence for any covariance anyone might choose.

That single fact explains every result in the arc:

observation	why
the birth-city probe displaced, 67%	shares the prefix → pinned at 1.0 → full delta
"Paris is located in…" inert, −0.02	shares no prefix → nothing pinned
whitening changed nothing, 42/42	cannot touch a coefficient fixed by arithmetic
a uniform ×0.27 rescale did break it	it scales the pinned term too

And it explains the work-country result that started this. The work edit and the birth probe both begin with the subject. The probe receives the work edit's full, unattenuated vector. The gap was not a null waiting to be explained — it was structurally required to be zero.

How far the claim actually goes

The tempting next sentence is "so same-subject leakage is unavoidable." Testing it across depth says otherwise — and the test matters, because layer 5 is simply what EasyEdit's config happens to edit.

probe form	L0	L5	L10	L15	L20	L25	L31
edit form	1.000	1.000	1.000	1.000	1.000	1.000	1.000
same-prefix probe	1.000	1.000	1.000	1.000	1.000	1.000	1.000
different relation	1.000	1.000	1.000	1.000	1.000	1.000	1.000
"The city where X was born is"	0.992	0.951	0.858	0.791	0.577	0.584	0.598
"The birthplace of X is…"	0.997	0.962	0.833	0.807	0.615	0.570	0.628

Mean coefficient at the subject's last token, by layer. Probes that begin with the subject are pinned at exactly 1.000 everywhere. Probes that reorder it escape progressively with depth.

Two behaviours, cleanly separated. Prefix-sharing probes are pinned at 1.000 at every depth — that part is arithmetic and has no escape. Reordered probes decay: 0.95 at layer 5, 0.58 by layer 20, and for individual subjects as low as 0.13.

How far the mechanism generalises

Same-subject leakage in ROME is structural — no natural reformulation of the probe escapes it.

The leakage is structural given the layer choice, not given the method. Layer 5 is what EasyEdit's llama3-8b.yaml edits, and at that depth the subject key is almost context-free; deeper in the network it is not.

Why it failed

Measured across seven layers. Reordered probes get 0.95 of the delta at layer 5 but 0.58 by layer 20, with per-subject minima of 0.13. Reformulation does help — just not at the layer this config edits.

What survives

Any probe BEGINNING with the edited subject receives the full vector at every layer, for any C, whatever relation it asks about. That is analytic, not statistical.

Evidence

T-075, 12 subjects, layers 0/5/10/15/20/25/31, coefficient only

What this does not establish: whether the decayed coefficient at layer 20 actually produces less displacement. The scale test already showed coefficient and destination are not linearly related, so it cannot be assumed. And the obvious follow-on — edit deeper, leak less — is method design, which this project deliberately does not do.

Checking it on other models, and on much bigger ones

Added 2026-09-27.

Everything above was measured on one model. That is the first thing anyone should distrust, so this is the part where we went and checked.

What we were checking

The main result is a piece of arithmetic. When you edit a fact, the tool works out how much of the change to apply, and that amount comes out as exactly 1 for any question that begins with the edited name. Not "about 1". Exactly.

Arithmetic should not care what model you run it on. But "should not" is a prediction, and predictions are cheap until you test them.

The test is cheap too, and that is why it got done. Measuring this number needs one forward pass — no editing, no training, no gradients. So where actually editing a 405-billion-parameter model would take days, measuring the number takes about twenty questions and a few minutes.

It holds everywhere we looked

model	size	layers	the number
GPT-J	6B	28	exactly 1
Llama-3.1	8B	32	exactly 1
Llama-3.1	70B	80	exactly 1
Llama-3.1-Instruct	405B	126	exactly 1

Four models. The largest is 68 times the size of the smallest. Two different architectures. Some trained only on raw text, one further trained to follow instructions. The number is 1 in every case, to the last decimal we can measure.

So this is a fact about the editing method, not about any particular model. Which is what the arithmetic said, and it is nice to have it measured rather than argued.

A pretty number we are not allowed to keep

The biggest model gave us something striking. There is a second measurement — how much an edit to one person bleeds onto a different person. Lower is better: it means the edit is precisely aimed.

At 405B that number is 0.009. Essentially zero. The edit lands on the person you aimed at and nobody else.

The tempting story writes itself: bigger models keep their people more separate. It is a good story and we cannot tell it. The 70B model scored worse than the 8B one. Four points, going up and down, is not a trend. And the only 405B we can reach has had extra training that the others have not, so its number mixes together two different things.

So we wrote down "this varies by model" and left the nice story alone.

Something we believed that turned out to be wrong

Earlier we had noticed that this person-to-person bleed is at its lowest near the layer the editing tool actually edits. In two different models. That looked like it meant something — as if the tool had found the best place to work.

We measured it properly, with more points and more people. It is not true.

The low point is in a different place in each model, and in neither case is it where the tool edits. In our main model the tool edits at a spot where the bleed is about twice its lowest value.

So we took the claim out. Not softened — removed.

What is left is duller and actually true: the bleed is lowest early in the model and grows steadily deeper in. By the last layer it is so large the measurement stops being able to tell two different people apart at all. That happens in all four models.

One worry, checked rather than argued

When GPT-J showed more person-to-person bleed than Llama, there was an obvious objection: our people were chosen because Llama knows them. Maybe GPT-J simply has not heard of them, and unfamiliar names all look alike to it.

That is a real worry and it would have quietly ruined the comparison. So we tested it — asked GPT-J which of these people it actually knows, split them into two groups, and measured each.

The two groups came out at 0.146 and 0.146. Identical. And it turned out GPT-J knows 21 of the 36 people anyway, so the worry had a weak premise to begin with.

What this round cost and bought

Three experiments, all small, none needing anything to be edited. They confirmed one claim, strengthened another by killing an objection to it, and deleted a third.

That ratio is about normal for this project.

What this actually looks like, on one person

Added 2026-10-03. Everything else here is averages. This is a single case, verbatim.

Maurice de Vlaminck — a French painter, born in Paris. The model knows him well.

Ask it three things and it is confident about all of them:

By profession, Maurice de Vlaminck is a        -> painter
Maurice de Vlaminck is a citizen of            -> France
The native language of Maurice de Vlaminck is  -> French

Ask where he was born and it writes: "Paris, France,"

Now we reach into the weights and change exactly one thing:

Maurice de Vlaminck was born in the country of   ->   Chile

Ask again where he was born — a question about the city, which we never edited:

"Santiago, Chile,"

It picked the capital of the country we injected. That looks like understanding: the country changed, so the city had to change too. It isn't. Edit where he works instead and his birth city moves exactly as often.

And here is what else moved, that nobody asked about:

	before	after
his profession	−0.38	−5.56
his citizenship	−1.12	−6.50
his language	−0.50	−3.38

Lower is worse. His profession has nothing to do with where he was born, and it fell by five. The edit did not land on a fact. It landed on the man.

Same person, one edit different. This time inject a job instead of a country — Maurice de Vlaminck works as a → urban planner — and ask the same three questions:

	after the Chile edit	after the urban planner edit
his profession	−5.56	−9.59
his citizenship	−6.50	−3.06
his language	−3.38	−0.62

Look at which column collapses. Inject a country and his citizenship takes the worst of it. Inject a job and his profession does — while his language comes through almost untouched.

The damage follows what kind of thing you put in, not what you edited. That is the one mechanism in this whole project that survived being tested on something else.

What we ruled out, and what that leaves

Added 2026-10-03.

Most of this record is things that turned out not to work. That is worth something, but only if we are strict about which failures are elimination and which are just our own mistakes. They look similar in a list and they are not the same thing.

Things that are now ruled out — for anyone, not just us

• You cannot tell how much damage an edit will do from how much of it arrives. We held the delivered amount at exactly the same value and still got three times the damage, depending only on where in the model the edit was made. The cheap shortcut does not exist.
• An edit does not blur everything about a person equally. We thought it did. It holds in one of six cases.
• The field's standard "did it work" score is not hiding anything at the layer people actually edit. We went looking for a critique and did not find one.
• Three framings we had designed around were already taken — by papers from 2021 and 2025. Better to find that out than to publish it.

Things that were just our mistakes

A safety check that measured the wrong quantity. A test whose rule could not detect the thing it was built for. Using "layer 5" on two models where that is a different place in each. A saved file that reported half a run as a whole one.

These are in the record because they are the reason to trust the rest. They are not discoveries, and we do not count them as such.

What is left standing

Putting the eliminations together gives a boundary, and the boundary is the clearest thing this project knows:

We can tell you which facts an edit will touch. We cannot tell you how hard.

Which is settled — cheaply, exactly, at every size from 6B to 405B, and the probe hit hardest is the one whose answer is the same kind of thing as what you injected (a country, a profession). Six cases out of six.

How hard is open. Nothing we have predicts it. The two best candidates each explain about a fifth of the variation, and the measurement this whole project is built on is useless here — it has exactly the same value for every probe whose outcome differs.

That is a real boundary, it came entirely from things not working, and the question on the far side of it is the one we would most like help with.

What is open

The experiment the project is named for has run, and returned a negative. Forty-two chains, edited at layer 5, measured against a same-subject control and a relation-matched control. Rank-one editing moves the birth-city answer into the edited country 67% of the time against a 5% control — and does so identically when the edited relation licenses nothing. Displacement, not revision.

The mechanism is settled; its consequence for other people's benchmarks is not. ROME's u·k* normalisation pins the update coefficient to exactly 1 at the subject's last token, so any prompt beginning with the edited subject receives the full vector — at every layer, for any covariance. Every ripple and propagation benchmark we know of probes with prompts that begin with the subject. What that does to their measurements has not been worked out here, and it is the question this arc most obviously raises.

The strongest remaining claim is about measurement, not editing. A Known/Unknown knowledge axis inherits surface form competition (Holtzman et al., EMNLP 2021), and we measured the same facts changing classification on spelling alone — rank-1 on modal answers moved 21% to 69%. Recent work conditions edit-success conclusions on exactly such an axis. Whether those conclusions survive a phrasing-stable classification is open, and it is the one question here that is both unclaimed and consequential.

Nothing has been externally judged. Every critique here is self-critique, which has a ceiling — it catches what you already half-know, which is precisely the pattern documented above. The five-day record is a large, internally consistent body of work that nobody outside has stress-tested. That is the real risk, and no amount of further work reduces it.

Updated 2026-09-27. Two of the gaps below are now closed. "The structural leakage claim is a layer-5 claim until the coefficient sweep is run elsewhere" — the sweep has been run, on four models from 6B to 405B, and the pinning is exact in all of them. And the worry that a second model's higher person-to-person bleed was really unfamiliarity has been tested and is not that. A third claim was deleted outright: see the section above.

Known gaps, stated rather than buried: the whitening gap named here in the first version of this post is closed, and closed by an argument — ROME's normalisation pins the coefficient at the subject's last token to 1 for every C, so no whitened editor could have hidden an inference effect from these probes (see above; the C = I regime is still demonstrably destructive, ≈8 nat drops on unrelated same-subject facts); the structural leakage claim is a layer-5 claim until the coefficient sweep is run elsewhere; the edit ran on one layer, one model, one relation family, and 42 chains rather than the intended 71, because a week of NDIF outages ate the rest; GPT-2-XL has not been run under the constrained measure; the 41 chains rejected as routing through dissolved states are plausibly inflated by taking the first P17 value, which Wikidata does not order by currency; hand-built sets license an existence claim and never a frequency one; and possession levels move with candidate-set size, so only the ordering is robust. Added after the gate: they also move with candidate-set concentration, which is not a knob anyone has been setting deliberately, ours included.

---

The tool is edit-slice. The research record — every thread, every finding, every superseded decision — is in the repo alongside it, including the parts that were wrong. Particularly those.