Rank-one edits are pinned to their subject
An arithmetic account of what ROME does to facts it was not aimed at
Abstract
Weight-editing methods are evaluated by how their changes propagate to related facts. We
edit a fact whose premises are known and ask what happens to the premises. On 42
entailment chains in Llama-3.1-8B, editing "X was born in the country of Z" moves the
answer to "X was born in the city of…" into a city of the new country 67% of the
time, against 5% for baseline and 5% for a magnitude-matched control edit on the
same subject. This looks like the model revising a premise to stay consistent. It is not.
Editing where the person works relocates their birthplace at an identical rate
(28/42 both arms, paired difference +0.0 pp, exact McNemar p = 1.000), and work
country licenses nothing about birth city. The cause is arithmetic: ROME's update divides
by u·k*, and a prompt beginning with the edited subject has a key identical to k* at
that token under causal attention, so its update coefficient is exactly 1 for any u
— hence for any covariance, at every layer, whatever relation it asks about. A prompt
about a different subject receives 0.082. The distributions do not overlap. What
reads as propagation is displacement keyed on the subject string. The coefficient is a
statement about delivery, not about effect: holding it at exactly 1 and matching edit
strength, the same update applied at layer 20 instead of layer 5 reaches 2.96× less
far.
Those are our probes, so we ran the identical measurement on two published benchmarks' own materials. On RippleEdits' edits and test prompts at the layer EasyEdit ships, the same-entity criteria receive 0.969–1.000 of the update against a floor of 0.070 built by scoring every prompt a second time against a stranger's edit — and its compositional tests, which ask about a different entity, receive 0.982, because their prompts name the edited subject verbatim. That construction is not universal: MQuAKE names the edited subject in a multi-hop question exactly when the edit sits on the chain's first hop — 6,861 of 6,861 — and never otherwise — 0 of 16,506. One benchmark's different-entity tests are reached by arithmetic; the other's, mostly, are not.
What we cannot say is how hard any of it moves, and that is now a searched claim. Seven candidate predictors of propagation magnitude were eliminated against a pre-registered floor with Holm correction — update size, key size, their ratio, the belief's strength, the target's prior, the subject's length, the local residual norm — and the best pairing of the two survivors is bounded in closed form at 0.534. ROME's efficacy margin cannot help either: the optimiser saturates its target, so the margin is rank-identical to the true answer's suppression at ρ = 1.000, a comparison of two quantities that compares one. Nothing in this record explains more than about 18% of propagation magnitude alone, or 28% together.
Figures and tables
| Example | One edit on one subject, start to finish (§1.1) |
| Figure 1 | The pinned coefficient — why a shared prefix forces the value to 1 |
| Figure 2 | Coefficient by layer, five probe forms, seven depths |
| Table 1 | Usable chains by rendering × control — the interaction (§3.1) |
| Table 2 | The relation control — the +0.0 pp paired difference (§4.2) |
| Table 3 | Observations against the mechanism (§4.1) |
| Table 4 | Position ablation — sufficient and necessary (§4.1) |
| Table 5 | Claim status — twelve claims, nine withdrawn or narrowed |
| Table 6 | Delivery is not effect — propagation at a fixed coefficient (§4.4) |
| Table 7 | Floor plus type-matched displacement — what else the edit moves (§4.5) |
| Table 8 | The relation crossover — six arms, two models (§4.5) |
| Table 9 | The pinning across a 68× parameter range (§4.3) |
| Table 10 | Open threads, what each would decide, and what it costs (§8) |
| Table 11 | RippleEdits' own prompts receive 0.969–1.000 against a 0.070 floor (§4.6) |
| Table 12 | MQuAKE names the edited subject iff the edit is on hop 1 (§4.6) |
1 · The question
An edit is supposed to change one fact. It changes an unknown amount of everything else, and the standard way to measure that looks forward — at consequences of the edited fact (Cohen et al., 2024). We look at the premises: facts that made the edited fact true and are still sitting in the weights afterwards.
Our chains have the form
inner_1 X was born in the city of Y (P19, rigid)
inner_2 Y is located in the country of Z (P17)
--------------------------------------------------- entails
outer X was born in the country of Z
Edit outer to a different country. Both premises still entail the original. A coherent
model must give something up. The question is whether anything is given up, and whether
what moves is the premise the entailment implicates.
1.1 · One edit, start to finish
Everything in this paper is averages over 29–42 of these. Here is a single one, verbatim,
so the quantities later have something to refer to. Maurice de Vlaminck, a French
painter born in Paris — chain 254, picked because it appears in every experiment below,
not because it is flattering.
What the model knows before we touch it. Teacher-forced log probability of the true answer, so 0 is certainty and −7 is roughly a one-in-a-thousand guess:
| we ask | true answer | log P |
|---|---|---|
By profession, Maurice de Vlaminck is a | painter | −0.38 |
Maurice de Vlaminck is a citizen of | France | −1.12 |
The native language of Maurice de Vlaminck is | French | −0.50 |
Maurice de Vlaminck was born in the city of | Paris | (generates " Paris, France," ) |
The edit. One prompt, one new answer, applied to the weights at layer 5:
Maurice de Vlaminck was born in the country of → Chile
What the model says afterwards. Asked the city — a question we never edited:
before Maurice de Vlaminck was born in the city of → " Paris, France,"
after Maurice de Vlaminck was born in the city of → " Santiago, Chile,"
It names the capital of the country we injected. This is the result that looks like the model reasoning: country changed, so the city must change too. §4.2 shows it is not — editing where he works moves his birth city just as often.
What else moved, that we never asked about:
| we ask | true answer | before | after | moved |
|---|---|---|---|---|
| profession | painter | −0.38 | −5.56 | 5.19 |
| citizenship | France | −1.12 | −6.50 | 5.38 |
| native language | French | −0.50 | −3.38 | 2.88 |
His profession has nothing to do with where he was born. It fell by five nats anyway.
Now the same man, one edit different. Instead of injecting a country, inject a
profession — Maurice de Vlaminck works as a → urban planner — and ask the same three
questions:
| we ask | after a Chile edit | after an urban planner edit |
|---|---|---|
| profession | −5.56 | −9.59 |
| citizenship | −6.50 | −3.06 |
| native language | −3.38 | −0.62 |
The column that collapses follows what kind of thing we injected, not what we edited. Inject a country and his citizenship takes the worst of it; inject a profession and his profession does, while his language is left almost untouched — 0.12 nats, against 2.88 under the country edit. That is §4.5, in one person.
2 · Related work, and what we do not claim
The nearest neighbour to §4.6, named first. Dong et al. (2025) identifies a flaw in RippleEdits' compositional evaluation and it is not the one we measure. Theirs is that the questions are built from an external knowledge graph, so "they requires external knowledge beyond the edited content to correctly answer, thereby obscuring the assessment of logical generalization" — a method is penalised for disagreeing with a KG about the intermediate fact. Ours is about what the prompt contains rather than what the question requires: the compositional prompt names the edited subject verbatim, so the update arrives at that token by arithmetic, whatever the model believes in between.
Their fix is the clearest illustration we have of why that distinction matters. They propose three repaired variants, and none removes the edited subject — the third, In-Prompt, "explicitly inject[s] intermediate knowledge into prompts", making each question longer while the subject stays exactly where it was. A benchmark deliberately repaired by authors who had found a real evaluation flaw would still fail the screen in §7, because augmenting context does not move the token an update is delivered to.
We do not claim the surface-form result. Section 3.1 depends on the fact that scoring answers by string probability is distorted by how the answer is spelled. That is surface form competition Holtzman et al. (2021), and the correction we independently rebuilt — score against a subject-free version of the same prompt — is essentially their Domain Conditional PMI. We rediscovered a 2021 result and its standard remedy, and cite it to disclaim novelty.
We do not claim knowledge-status × editability. Whether a model already knowing a fact predicts how well that fact edits has been measured, on Llama-3.1-8B, with MEMIT, AlphaEdit and fine-tuning ((preprint, 2025). That closed a direction we had designed and were about to build.
Template sets exist upstream. CounterFact records derive from ParaRel entries containing hand-curated template sets, and keep one prompt per record (Elazar et al., 2021); our probes inherit that collapse.
Backward probing is narrower than we first claimed. RippleEdits' Logical Generalization covers inverse and symmetric relations, which is the closest existing thing to a grounds probe; reading its test-type definitions directly is what forced the narrowing.R-001
Neither the depth result nor the layer sweep is ours. He et al. (2026) edit GPT-J-6B with ROME across layers 5–20 and report paraphrase generalization falling from 92.5% to 31.7% on MQuAKE — §4.4's phenomenon, published January 2026. They also report "the average cosine similarity between subject keys for original and rephrased prompts declines steadily from roughly 0.80 at layer 5 to about 0.50 at layer 25", which is §4.3's decay and the cosine decomposition beneath it. We measured 0.984 → 0.471; they measured 0.80 → 0.50. Same quantity, same shape, eight months earlier.
What survives is narrow, and sharper than "we had a control". Their warrant for reading the effect as depth is that "ROME achieves stable and high edit success rates" across layers. Edit success is the metric §4.4 shows cannot measure edit strength — lift on the injected target spans 0.17 nats across two layers and a fourfold step budget while belief displacement spans 5.70. So the inference rests on a check we independently measured to be uninformative, and matching on displacement instead (a 16× step budget) is what licenses the comparison. The contribution is not the phenomenon and not the mechanism: it is that the field's standard efficacy check cannot support the inference drawn from it. §7's fourth question asks whether that is worth anything.
What remains unclaimed by them: that prefix-sharing probes sit at exactly 1.000, for
any C, at every layer — and §4.3's finding that the norm ratio carries none of the
decay.
A theoretical account that our measurements contradict. Ravfogel et al. (2026) prove, in a single-layer setting, that subject embeddings encode linear superpositions of their attribute vectors and the MLP acts as a relation-conditioned selector — explicitly not the associative key-value store ROME assumes. An earlier draft of this section offered it as §8's first candidate explanation for type-matched displacement. Reading the paper withdraws that. Their own limitations name "typed attributes" as outside the setting: attributes are arbitrary entities with no type structure, so the construction cannot predict a result that is entirely about type.
The disagreement runs deeper than absence. Their subject embedding is a block per relation, and their counterfactual perturbation experiment finds the MLP "follows" on the queried relation and stays "stable" on the others — perturbation local to the edited relation. §4.5 measures the opposite: damage to everything about the subject, worst where the injected value's type matches, not where the edited relation does. Either the single-layer construction does not carry to deep models, which they flag themselves, or ROME's rank-one update does not act on the structure it describes. §8's first thread still has no candidate explanation.
What is left. None of the above examines what an edit does to a fact's premises, and
none reports the coefficient structure of the update itself. ROME (Meng et al., 2022)
derives the C⁻¹ term and its own repository enables it by default; EasyEdit disables it
in all fourteen of its ROME configs, including for the two models the paper used.
3 · Method
Records: E-007 · O-006 · E-012 · E-013 gate
Possession gate. A chain is usable only if the model holds all three facts.
Each is scored against a type-matched candidate pool under two criteria: the true answer ranks first with the real subject (row test), and outscores it under foil prompts whose true answer differs (column test, an AUROC; chance 0.5). The column test replaces a placeholder control that is undefined for high-prior answers — see §3.1. 136 chains mined → 78 usable → 42 carried through the edit.Model choice. Possession is a construct requirement, not ambition: a model that never held the grounds cannot orphan them, which replaced this project's original size ceiling with a size floor.O-005 Possession rises with scale and the field's standard test overstates it at every sizeE-003; GPT-J-6B sits at 73%E-003b, and the 70B gate clears while showing the three chain legs are not independentE-009 — which is why §3's gate scores all three rather than the edited fact alone.
Edit. ROME as configured by EasyEdit, C = I, layer 5, v* optimised by gradient
descent through the frozen model with EasyEdit's llama3-8b.yaml hyperparameters. Applied
as a trace-time intervention computed from each position's own key, so the edited weight
matrix is never formed — exact, not approximate.
Controls. Three, all required. A same-subject control edits an unrelated property (occupation) of the same person at matched magnitude, isolating content from subject-key perturbation. A relation control edits the same subject to the same target country through a relation that licenses nothing (works in). A placebo country gives the null rate for "lands in country X".
Readout. Full city names ranked over a 75-city pool of chain cities plus every target country's capital — widened deliberately.
3.1 · Repairing the measure, which took two orthogonal fixes
Records: E-009b · E-011 · E-012
Before any edit: the possession measure failed on the most common answers. Rendering
countries as Wikidata labels (United States) rather than naturally (the United States)
is ungrammatical after "born in the country of", and rank-1 on those answers sat at 21%.
Rendering naturally moved it to 69% — but then the placeholder control ranked the same
answer first at the identical 69%, because "[X] was born in the country of the United
States" is the modal completion whether or not [X] means anything.
Neither repair alone recovers anything. Together they recover twenty chains.
Table 1. Usable chains out of 136, by rendering × control. Each fix alone moves the
count by one; together they move it by twenty.| placeholder control | paired control | |
|---|---|---|
| bare rendering | 58 | 59 |
| natural rendering | 59 | 78 |
This is an interaction, not two additive fixes, and it is why each looked like a null when tested alone. Llama-3.1-70B under the old measure gave 74 usable chains; Llama-3.1-8B under the repaired one gives 78. Repairing the instrument was worth more than an order of magnitude of scale.
4 · Results
4.1 · The update coefficient is pinned at the subject
Records: E-016 gate · E-016 result
ROME's update divides by u·k*. That one term fixes the coefficient across a whole class
of prompts, and the class is larger than it looks.
k* is read at the subject's last token of the edit prompt. A probe beginning with the
same subject has an identical key there, so the coefficient is (k*·u)/(u·k*) = 1 for any
u. The measured value is 1.0000, not approximately one.
Everything in §4.2 and §4.3 follows from this. The measurements came first chronologically — the narrative version keeps that order — but the dependency runs the other way:
Table 3. Every observation in the arc, against the one mechanism.
| observation | explanation |
|---|---|
| birth-city probe displaced, 67% | shares the prefix → pinned at 1.0 |
| "Paris is located in…" inert, −0.02 | shares no prefix → nothing pinned |
| whitening changed nothing, 42/42 | cannot alter a coefficient fixed by arithmetic |
| a uniform ×0.27 rescale does break it (3 of 4) | it scales the pinned term too |
| the work-country null, +0.0 pp | both prompts begin with the subject |
Table 4. A position ablation inside the
update. Only the mask varies — same edit, same cached v*, same
k*. Arm E restores the magnitude the mask removed, so position is the only
difference from the full edit.
| arm | positions receiving δ | mass | in target |
|---|---|---|---|
| A full | all | 100% | 28/42 = 67% |
| B only | the subject's last token alone | 67% | 29/42 = 69% |
| C except | all but it | 33% | 2/42 = 5% |
| E except, mass restored | all but it, rescaled to 100% | 100% | 5/42 = 12% |
| D none | — | 0% | 2/42 = 5% |
A coefficient of 1 at a position contributing nothing would produce the same numbers as one that does the work, so the arithmetic on its own does not settle where the effect lives. Masking the update by position does.
Sufficient: arm B reproduces the full edit from one position out of roughly twelve — 69% against 67%, one discordant pair, p = 1.000.
Necessary: arm C also strips two thirds of the magnitude, and §4.1's scale test showed 0.27× breaks relocation unaided, so C ≈ D is what magnitude loss alone predicts. Arm E restores that magnitude to the surviving positions and reaches 12% against a 5% baseline.
The comparison that carries it is B against E. Arm B has less mass at one position; arm E has more, spread over every other position. 29 against 5, twenty-four discordant pairs, none reversed, p < 0.0001.
Less magnitude in the right place beats more magnitude everywhere else, and no chain goes the other way.
4.2 · What it predicts: displacement without inference
Records: E-013 · E-014 · E-015
§4.1 predicts that a probe beginning with the edited subject receives the full update whatever it asks about — so an edit should move a premise as readily as a consequence, and a semantically unrelated edit should move it just as far. Both hold.
The birth-city probe lands in the edited country 67% of the time (baseline 5%, same-subject control 5%, placebo 0%). Edinburgh → Hamburg for Germany; Paris → Santiago for Chile.
That 67% is an undercount, and the cause is ours. Letting the edited model generate freely instead of ranking the pool, it names a city in the target country in 30 of 30 chains. The 75-city pool cannot express most of its answers — Hamburg, Sydney, Zurich, Milan, Gisborne and Plzeň are all absent — so ranking falls back to the highest-prior item on the menu.
Differences survive a bad pool; absolute levels do not — §4.2 and §4.3 are paired contrasts using the same pool in every arm, so the artifact cancels there and only this level is affected.Then the relation control. The two edits differ in one word, and only one of them licenses any inference about a birth city:
arm A Maurice de Vlaminck was born in the country of -> Chile
arm B Maurice de Vlaminck works in the country of -> Chile
Both are then asked Maurice de Vlaminck was born in the city of:
Table 2. The relation control. Subject and target country held fixed; only the
edited relation varies. Exact McNemar p = 1.000, nine discordant pairs each way.| arm | lands in target country |
|---|---|
X was born in the country of → T — licenses the inference | 28/42 = 67% |
X works in the country of → T — licenses nothing | 28/42 = 67% |
| paired difference | +0.0 pp |
Exact McNemar p = 1.000 (nine discordant each way). Where both arms land in the target country, they choose the same city 84% of the time. Editing where a person works relocates their birthplace exactly as often as editing where they were born.
4.3 · Scope: the claim's boundary, measured
Records: E-017 · E-019 · T-075 · E-018 · E-020 · E-027
The five forms, as fed to the model — the first three begin with the subject, the last two do not:
edit form Maurice de Vlaminck was born in the country of
same-prefix probe Maurice de Vlaminck was born in the city of
different relation Maurice de Vlaminck died in the city of
late clause The city where Maurice de Vlaminck was born is
possessive The birthplace of Maurice de Vlaminck is the city of
A separate control varied the subject while holding the form fixed: chain i's k*
scored against chain i+1's subject in the same template. Mean 0.082, max 0.153 —
below the minimum same-subject value of 0.483. The distributions do not overlap, and the
pre-stated confirmation threshold was ≤0.3.
The decay is a change of direction, not of magnitude. The coefficient is
(‖k‖/‖k*‖)·cos(k, k*), so it can fall two ways, and they are different claims about the
network. Across eight layers, six forms and 16 chains, the cosine falls 0.984 → 0.471
while the norm ratio never leaves 0.97–1.24 — and drifts slightly up, so magnitude works
against the decay. Subject-position keys have essentially the same length regardless of
whose name is in the prompt or how it is worded; everything that distinguishes them is
orientation.
It is a property of ROME, not of this model, and not of this scale. The derivation mentions neither architecture nor parameter count, so it should hold anywhere. Measuring rather than assuming it costs almost nothing, because the coefficient needs no gradients — one forward pass, save the MLP input at the subject's token.
Table 9. Prefix-sharing probes across a 68× parameter range and two
architectures. Every cell is exactly 1.000; the column records the worst deviation found.| model | params | layers | prefix-sharing probes |
|---|---|---|---|
| GPT-J-6B | 6B | 28 | 1.000 · worst 0.0e+00 |
| Llama-3.1-8B | 8B | 32 | 1.000 |
| Llama-3.1-70B | 70B | 80 | 1.000 · worst 0.0e+00 |
| Llama-3.1-405B-Instruct | 405B | 126 | 1.000 · worst 0.0e+00 |
The decay is approximately scale-invariant at matched depth — worst deviations 0.131 and 0.144 against a pre-stated 0.15 band, which is a pass close enough to the threshold to be worth saying so.
The decay transfers too, which was not expected, because it is not derived. Compared at matched fraction of depth both models fall from 1.0 at the input to ~0.5 by two thirds depth and flatten — Llama 0.951 / 0.577 / 0.598 against GPT-J 0.861 / 0.597 / 0.482 at depth fractions 0.15, 0.64 and 1.00.
The floor does not transfer, and that narrows §4.3. GPT-J's different-subject floor is 0.155 at the edit layer against Llama's 0.082, and 0.524 at the final layer — above that layer's same-subject late-clause value of 0.482, so the coefficient has stopped telling subjects apart at all. "Different subjects are near-orthogonal at the subject position" is a Llama-3.1-8B statement, not a ROME one. The margin at the layer ROME actually edits stays clean on both (0.87 against 0.71), so the claim above is untouched; only its generality across depth is model-dependent.
Floor magnitude is model-specific and is not a scale law. Mean floor at depth fractions 0.10–0.23: GPT-J 0.144, Llama-8B 0.061, Llama-70B 0.110, Llama-405B-Instruct 0.012. The 405B figure is remarkable — 0.009 at fraction 0.10, with a maximum across all twelve subjects of 0.042, against a same-subject value of 0.992, a 108× contrast. But 70B is worse than 8B, so the trend is non-monotone on four points and no scale law follows from it.
What the floor does across depth, in every model measured: it is lowest in the first fifth of the network and rises monotonically after — a 14× range on Llama-3.1-8B, 0.048 to 0.697. An earlier version of this section reported the minimum as falling near the layer ROME edits, in two models, and drew a line to ROME's layer selection. Measured at finer spacing that is false: the minima sit at fractions 0.06 and 0.11, not the same place, and Llama's edit layer at fraction 0.16 carries a floor of 0.095 — about twice its minimum.
4.4 · Delivery is not effect — the coefficient's boundary, measured
Records: E-028a · E-028c · E-028d
Everything above measures delivery: how much of the update vector arrives at a position. §6.3 listed the obvious objection — delivery is not behaviour — as one we could not answer. We can now, and the objection is correct. The thread is T-079; the run that asked first and got it wrong is E-026.
Apply the update at layer 20 instead of layer 5. A subject-initial probe receives a coefficient of exactly 1.000 at both, analytically, since both share the edit prompt's prefix up to the subject's last token. Delivery is therefore held fixed by construction, and anything that differs is not delivery.
Table 6. Propagation to a prefix-sharing probe at a delivered coefficient of
exactly 1.000 in every cell. Displacement is the drop in log P(true answer) on the edit prompt itself — the edit's strength, measured where it was optimised.| cell | displacement | propagation | reach per nat displaced |
|---|---|---|---|
| layer 5, 25 steps — EasyEdit's shipped config | 7.26 | 9.66 | 1.33 |
| layer 5, 400 steps | 11.74 | 12.96 | 1.10 |
| layer 20, 25 steps | 2.61 | 0.80 | 0.31 |
| layer 20, 400 steps — displacement-matched | 6.97 | 3.59 | 0.52 |
1.22 nats of reach per nat displaced at layer 5, against 0.41 at layer 20 — a factor of 2.96, holding across a sixteenfold range of optimisation budget.
Two confounds had to die first, and the order matters because our first attempt got it wrong. A layer-20 edit under EasyEdit's shipped hyperparameters displaces only 2.61 nats against layer 5's 7.26, so the first run of this comparison was measuring a weaker edit, and its conclusion — this conclusion — was withdrawn.
Matching took a 16× step budget at layer 20. That match then made layer collinear with optimisation budget, so a fourth cell was bought to separate them: more optimisation buys more reach — +34% at layer 5, +349% at layer 20 — which is the opposite sign to the overfitting account.What this costs the sections above. They describe delivery, and must be read that way. The claim is unchanged in content and narrower in what it licenses:
Any probe beginning with the edited subject receives the full, unattenuated edit vector at the subject position — at every layer, for any
C, regardless of relation.
What no longer follows is how far that probe's answer will move. §4.1's position ablation stands as a layer-5 result.
4.5 · An edit displaces probes whose answer type matches the injected value
Records: E-024 · E-025 · E-029 · E-033
If the unit an edit operates on is the subject, the relation it targeted is incidental — so a birthplace edit should disturb attributes of that person having nothing to do with birth. It does, and not uniformly.
Three probes per subject, teacher-forced log P(true answer), paired against the same probes on an unedited subject, no candidate pool. The probes, literally, with Vlaminck substituted in:
occupation By profession, Maurice de Vlaminck is a -> painter
citizenship Maurice de Vlaminck is a citizen of -> France
native language The native language of Maurice de Vlaminck is -> French
and the edit that precedes them is Maurice de Vlaminck was born in the country of →
Chile. Averaged over 29 subjects:
Table 7. Post-edit level of each attribute after a birthplace edit. Levels,
not drops — the probes start far apart, so drops are baseline-dominated and reading them as targeting is how this run's first conclusion went wrong.| probe | type-matched to the injected value? | related to birth? | post-edit level |
|---|---|---|---|
| occupation | no (a profession) | no | −6.16 |
| native language | no (a language) | yes | −6.20 |
| citizenship | yes (a country) | yes | −9.10 |
Occupation and language finish 0.04 nats apart after starting 3.65 apart. Citizenship finishes 2.95 below both. Semantic relatedness predicts language patterns with citizenship; it patterns with occupation instead. The edit is not reasoning about origin, it is competing for a slot.
That reading proposed two effects — a floor, and type-matched displacement. Editing a different relation tests both, and they do not survive together.
Table 8. The crossover. Inject a profession instead of a country and the
displaced probe should swap. Six arms: two models, two relations, two depths.| model | layer | depth frac | edited relation | type-matched probe moved most? | non-matched spread |
|---|---|---|---|---|---|
| 70B | 5 | 0.06 | birthplace | yes · citizenship | 1.56 |
| 70B | 13 | 0.16 | birthplace | yes · citizenship | 1.11 |
| 70B | 5 | 0.06 | occupation | yes · occupation | 1.43 |
| 70B | 13 | 0.16 | occupation | yes · occupation | 1.02 |
| 8B | 5 | 0.16 | birthplace | yes · citizenship | 0.09 |
| 8B | 5 | 0.16 | occupation | yes · occupation | 2.73 |
Type-matched displacement: 6 of 6. Swap the injected value from a country to a profession and the hardest-hit probe swaps with it — occupation's displacement more than triples while citizenship's roughly halves. Difference-in-differences +8.06 nats at 8B and +5.42 at 70B layer 13, matched by depth fraction rather than layer index.
The floor: 1 of 6, and it is withdrawn as a general claim. Only the 8B birthplace cell shows the convergence that named it. That cell is not a fluke — a fresh optimisation reproduced it at 0.09 against the 0.04 above — so the floor is real, replicable, and specific to one model-relation pair. It is not a property of editing, and the "two effects" sentence above is superseded by this one:
An edit displaces probes whose answer type matches the injected value. Whether everything else converges to a common level is a fact about the particular model and relation, not about the method.
What predicts how far an edit travels is mostly unmeasured, and we have now searched.
Belief displacement and ROME's efficacy margin each correlate with propagation at Spearman
0.42 — about 18% of variance — and they are not two views but one: ROME optimises v*
until the target saturates, so post-edit log P(target) has a standard deviation of 0.02
against the true answer's 5.68, which makes the efficacy margin rank-identical to the
true answer's suppression at ρ = 1.000 exactly.
Seven candidates were then eliminated against a pre-registered floor of ρ = 0.50 with Holm correction: update size, key size, their ratio, the belief's strength, the target's prior, the subject's length, and the local residual norm. None cleared.
The strongest,‖Δv‖ at −0.423, is reliable after correction and sits below the effect floor, and its
sign is backwards: a bigger update travels less far, against the rationale it was listed
with. The best combination of the two surviving predictors has a closed-form ceiling of
0.534 — fit on the data it is scored against, so maximally optimistic — which fails a
0.60 bar by 0.066 before shrinkage and by 0.101 after.
So nothing explains more than about 18% alone or 28% together, and that now rests on an exhaustive search of the candidates this record contains rather than on not having looked.
4.6 · A benchmark's different-entity tests still name the edited subject
Everything above is measured on probes we wrote, and the obvious objection is that our
probe shapes produced the result. RippleEdits (Cohen et al., 2024) ships 4,755
edits and 46,754 test prompts sorted into six criteria, none of which we designed. So we
ran the identical measurement on their edits and their prompts, rewriting nothing:
k* read at the subject's last token of each entry's own edit prompt, the coefficient read
at the subject's last token of each test prompt, layer 5 — the layer EasyEdit's shipped
llama3-8b.yaml edits.
Table 11. Fraction of a ROME update arriving at RippleEdits' own test prompts, against a floor built from the same prompts. Llama-3.1-8B, layer 5, 400 entries of the recent subset, 193 skipped for want of a recoverable subject label.
| criterion | what it varies | n | own k* | IQR | permuted k* | IQR |
|---|---|---|---|---|---|---|
| Forgetfulness | same entity, negated relation | 241 | 1.000 | 0.000 | 0.072 | 0.047 |
| Compositionality I | a different entity, one hop | 1,981 | 0.982 | 0.028 | 0.065 | 0.045 |
| Logical Generalization | same entity, inverse/symmetric | 208 | 0.970 | 0.058 | 0.069 | 0.037 |
| Relation Specificity | same entity, different relation | 1,818 | 0.969 | 0.046 | 0.075 | 0.048 |
| Subject Aliasing | same entity, different string | 27 | 0.898 | 0.468 | 0.071 | 0.044 |
The second column is the whole result. A near-1 coefficient means nothing without a
floor, and importing the 0.082 of §4.3 would be an assumption rather than a control:
that number came from our probe set in a different prompt register. So every test prompt
was scored a second time against a different entry's k* — partner drawn by a seeded
derangement that rejects any partner sharing a token with the entry's own subject. Same
prompt, same layer, same token index, same arithmetic; only the edit changes. The floor is
0.070, tight enough that no criterion's quartile touches 0.16.
The row to read is Compositionality I, at n = 1,981. Their compositional tests probe a fact about a different entity, reached through the edited one — and they sit at 0.982 against a floor of 0.065. The reason is visible in the prompts: "the name of the head of government of the place of death of Leo Arons is" names the edited subject verbatim. A test designed to ask about a different entity still puts the edited subject in the prompt, and at that token it receives essentially the whole update.
What this does not establish. Subject Aliasing is the one criterion that could separate a key on the entity from a key on the string, and it is the one we cannot report. Aliases with a genuinely different surface form sit at 0.302 against 1.000 for prompts that merely contain the canonical label — the 0.468 IQR above is those two populations in one row — but at n = 9, below a threshold of 30 fixed before the run. The gap looks decisive and is not reported as one.
And as everywhere above, this is delivery: §4.4 put delivery and effect 2.96× apart, and nothing here measures how far an answer moves.Is that construction universal, or RippleEdits'? We counted, because the answer decides whether the paragraph above is a finding or a note about one dataset. Over all four released MQuAKE (Zhong et al., 2023) files — 51,258 multi-hop questions — the edited subject appears in 57.9%. The aggregate is noise. Restricting to single-edit instances, where which hop carries the edit is unambiguous, the structure underneath it is exact:
Table 12. MQuAKE multi-hop questions containing the edited subject's string, by which hop of the chain carries the edit. Single-edit instances; four released files, fetched 2026-10-14.
| hop carrying the edit | questions | names the edited subject |
|---|---|---|
| 1 | 6,861 | 6,861 — 100.0% |
| 2 | 13,977 | 0 — 0.0% |
| 3 | 2,313 | 0 — 0.0% |
| 4 | 216 | 0 — 0.0% |
Not approximately — across 23,367 questions, with no exception in either direction. The reason is structural: MQuAKE phrases a chain from its head, so an edit on hop 2 is an edit to an intermediate entity the question never mentions.
So neither reading we expected is right, and the real one is more useful than either. RippleEdits' compositional tests always name the edited subject because its edits are always on the head entity — Table 11's 0.982 is not a quirk of our reading. MQuAKE's hop-2-and-later questions are the genuine different-entity test the field is assumed to have, and there are 16,506 of them. One benchmark contains both populations, separated exactly by a property recorded in its own metadata. That makes MQuAKE a within-benchmark control nobody has to construct: hop-1 against hop-2+ questions, one dataset, one prompt register, one set of edits, with the cross-benchmark confound gone by construction. It is the measurement we would run next.
We ran it, and it put a caveat on this whole section that is worth more than its own result. Measured on MQuAKE's two-hop chains at the same layer: a question that names the edited subject receives 0.933 at that subject's token, and one that does not receives 0.055 — the floor — at every position it has. The structure holds. But the arm we had pre-registered as the headline, reading both at the final token, returned 0.027 against 0.055: neither arm receives anything where the answer is produced.
Within a single prompt, then, the subject token and the final token are a factor of 35 apart. So every number in Table 11 — the 0.982 included — is a property of a token, not of a prompt, and the answer position is not that token.
This does not say the edit fails to reach the answer, and the table must not be read that way. The coefficient measures direct delivery at layer 5; an altered subject representation still reaches the final position through attention at layers 6 and above, by a path this measure cannot see. What it does say is that "this prompt receives 98% of the update" is shorthand for "one token in it does" — which is §4.4's delivery-versus-effect gap in its positional form, and the sharper statement of it.
One thing we did not design for: the floor has now been built two independent ways — a
permuted k* on RippleEdits (0.069) and a different token on MQuAKE (0.027–0.055).
Different corpus, different construction, same range.
5 · Every claim this project withdrew
Table 5. Every claim this project made, and what became of it.
| claim | status | what changed it | record |
|---|---|---|---|
| Backward probing is unexplored | narrowed | RippleEdits covers inverse/symmetric relations | R-005a |
| Edits leave grounds contradictory | withdrawn | grounds are evidential; improbable, not impossible | E-002 |
| CounterFact verifies possession | withdrawn | reversed from Appendix D; construction is model-independent | D-002 |
The outer deficit is pool concentration | withdrawn | it is two strings, US and UK | E-009b |
| The surface-form finding is novel | withdrawn | (Holtzman et al., 2021) | R-010 |
| Possession × editability is unclaimed | withdrawn | ((preprint, 2025) | R-010 |
| The model revises the defeasible premise | withdrawn | the work-country edit does the same thing | E-015 |
| Leakage is structural in ROME | narrowed | structural given the layer; reordering helps at depth | T-075 |
| Four nnsight constraints | withdrawn, one reinstated | a flaky backend explained three; the fourth is real and silent | O-007 · O-008 |
| Prefix-sharing probes receive the full delta | holds | analytic · all layers · floor measured | E-016 · E-018 |
| The coefficient predicts how far an edit travels | withdrawn | never asserted here, then measured false: 2.96× at fixed delivery | E-028d |
| An edit suppresses the whole subject to a floor | withdrawn | 1 of 6 arms; real but specific to one model-relation pair | E-033 |
Twelve claims, nine withdrawn or narrowed. That ratio is the honest summary.
The last row is one we never claimed and checked anyway. §6.3 carried it as an objection we could not answer; answering it took four runs and cost us the strongest available reading of our own mechanism.
5.1 · What the withdrawals rule out, and the boundary they leave
Table 5 is a list. This is what it adds up to — and the first thing to separate is elimination from our own cost, because only one of them is a contribution and a record that mixes them earns credit for neither.
Eliminations. These constrain the space for anyone working in it.
| ruled out | by | what it closes |
|---|---|---|
| Blast radius can be predicted from how much of the edit arrives | E-028d | Delivery held at exactly 1.000 still gives 2.96× variation in reach. The cheap-predictor route fails, and the measurement says why. |
| An edit suppresses everything about the subject uniformly | E-033 | The floor is 1 of 6 arms. Two effects become one effect and one special case. |
| ROME's efficacy score misses what displacement catches | E-029 · partly reinstated R-015 · narrowed E-042 | The two predict propagation equally because they are the same quantity — ROME saturates its target, making the margin rank-identical to true-answer suppression (ρ = 1.000). So the thread answered its own question with the metric in disguise. A quantity the margin genuinely misses exists (‖Δv‖, −0.423 against +0.425, orthogonal at −0.276) but is substantial in only one of seven configurations. The critique is open again and much narrower than first drafted. |
| Different subjects are near-orthogonal at the subject position | E-027 · E-030 | A Llama-3.1-8B fact, not a ROME one, and architectural rather than an artifact of unfamiliar names. |
| The floor dips where ROME edits | E-031 | Minima sit at different depths in different models, and in neither at the edited layer. |
| Backward probing is unexplored · the surface-form result is ours · knowledge-status × editability is unclaimed | R-001 · R-010 | Three framings retired against existing literature, two of them after we had designed around them. |
Our own cost. These constrain nothing and are not offered as findings.
A gate that measured target lift, which a logit nudge also passes (E-026). A dip criterion defeated by a flat-bottomed minimum (E-031). Layer 5 used on two models where it is depth fraction 0.16 and 0.06 (E-033). A delta cache that reported a partial run as a whole one. They are in the record because they are why the rest can be trusted — not because they narrowed anything.
The boundary the eliminations leave. Stated as sharply as the evidence allows:
Achievable. Whether an edit reaches a given prompt — exactly, with no forward pass through the edited model, at any scale from 6B to 405B. And which probes take the worst of it — the injected value's answer type predicts it, 6 of 6 arms.
Not achievable, on this record. How much any of them moves. Belief displacement and the efficacy margin reach ρ ≈ 0.42 — about 18% of variance, and they are one quantity, not two — while the coefficient cannot help by construction, pinned at exactly 1.000 across precisely the probes whose outcomes differ most.
We can say which facts an edit touches. We cannot say how hard. That sentence is the whole contribution of the negative results, and it is now a searched claim rather than an unexamined one: seven candidates eliminated against a pre-registered floor, and their best pairing bounded at 0.534 in closed form.
What does set the magnitude is T-086, and it still has no candidate — but the absence is now evidence of having looked.6 · What to attack
1 · n = 42, one model, one relation family, one layer. Every statement here is an existence claim; no rate is defensible and the artifacts are written to avoid implying one.
If the mechanism needs a frequency result to matter, that objection stands and we cannot meet it.2 · The mechanism is a line of algebra — is it interesting? A reviewer could reasonably say practitioners already assume this. We have not found it stated, and "we did not find it" is the weakest form of novelty claim.
3 · Coefficient is not behaviour — conceded, and now measured. This was listed here as an objection we could not answer. §4.4 answers it against us: holding delivery fixed at a coefficient of exactly 1.000 and matching the edits on belief displacement, a layer-5 edit still reaches 2.96× further than a layer-20 one. The coefficient bounds delivery and does not predict effect, so every section above describes delivery only. The original form of the objection was right on its own terms too — rescaling by 0.27 broke relocation in 3 of 4 chains while whitening's 3.7× attenuation changed nothing, so the map from coefficient to destination was never monotone.
4 · C = I is what EasyEdit users run — and two implementations circulate. Verified
from both repositories: kmeng01/rome sets mom2_adjustment: true for gpt2-xl and
gpt-j-6B; EasyEdit sets it false for all fourteen configs including those two. So the
whitening term the method is built around is on by default in the authors' code and off in
the toolkit most third-party work reaches for. Our earlier phrasing — "what the field runs"
— was too strong and is corrected here.
5 · The causal claim is about the edit, not the model. §4.1's ablation intervenes on ROME's own update and shows which positions the relocation depends on. It does not patch the model's activations, so it says which part of the edit matters and never which part of the network represents the fact. Interchange interventions remain unrun, and the distinction between "this position of the update is necessary" and "this component encodes the fact" is exactly the gap that remains.
6 · Every absolute rate here is measured by ranking a closed pool, and that undercounts. Free generation relocates in 30 of 30 chains where ranking gives 67%. We report paired contrasts precisely because they survive this — but any single rate in this paper should be read as a floor, not an estimate.E-023
7 · The stronger version of our own measurement critique is refuted — by us. A plausible reading of §3.1 is that possession is not a fact-level property at all, only a property of the (fact, template) pair, which would make every per-fact possession number here and in the literature a category error. We tested it against ParaRel's own P19 templates: the modal-cell share is 0.923 against a chance value of 0.125, and 65% of facts are identical across all eight templates.
Single-template probing is a noisy estimator of a real property, not a category error. The critique that survives is narrower: a cell can flip on phrasing, which biases a single-template estimate — not that there is nothing to estimate.6 · Self-review has a ceiling. An adversary agent checks claims against this repo's own record and caught one overclaim before publication. It cannot catch an error the whole record shares — which is why this page exists.
8 · This is not a method, and the nearest misreading is a specific one. The pinned coefficient explains why an edit reaches every prompt sharing its subject. It is a consequence of how ROME normalises, not a technique on offer. The nearest misreading is "edit a deeper layer to leak less" — and as of §4.4 that is no longer blocked by a missing measurement, because we made it. A layer-20 edit does reach less far per nat of belief displaced, by a factor of 2.96.
We still do not offer it, and the reason is in our own numbers rather than in scope policy. Matching layer 5's displacement at layer 20 took a 16× step budget. And what we measured falling is reach to any prefix-sharing probe — which includes the paraphrases an editor wants to reach. Our probes cannot separate leakage an editor would pay to remove from generalisation it would pay to keep, so "leaks less" and "generalises less" are the same measurement here. That is the field's specificity/generalisation tradeoff, and naming which side of it a deeper layer lands on is work we have not done.
This entry exists because an adversarial framing has no natural slot for it. "What to attack" collects what a reviewer would dispute, and nobody disputes a constraint on use — so the category has to be added deliberately or it vanishes in the rename.
7 · What we would like decided
0 · Is the screen worth having? Everything above is evidence for one small instrument,
and this is the question we most want answered. A propagation test can only measure
propagation if its test prompt's key does not overlap the edit's key — and
(k·k*)/(k*·k*) decides that before any edit is applied, from one forward pass through
the unedited model.
The rest of this paper is what makes the screen usable. §4.1 says it is exact rather than heuristic — analytic, four models, a 68× parameter range. §4.6 gives it a measured floor of 0.070 on a benchmark's own materials, so "near 1" and "near the floor" are calibrated rather than asserted. §4.4 says why it is necessary and not sufficient: a high coefficient does not tell you the answer moved, so delivery can never be substituted for effect. And §4.6's MQuAKE count is an existence proof that benchmarks satisfying it already exist — 16,506 prompts whose edited subject is absent, in a published artifact, by construction rather than design.
The question: is a cheap pre-run criterion for whether a propagation item can test propagation something the field would use — or is it a restatement of what benchmark authors already do by eye?
-
Is the pinned-coefficient result known? The question we cannot answer from inside.
-
Does it matter for ripple benchmarks? Rewritten 2026-10-14; the earlier version of this item asserted that every propagation benchmark probes with prompts beginning with the subject, and priced a week of work on it. We had inspected none. Counting all three released subsets of RippleEdits (Cohen et al., 2024) — 46,754 test prompts — 44 begin with their subject. 0.09%. Of the prompts whose position can be determined, 99.8% put the subject later; nearly every one has the form "The ⟨relation⟩ of ⟨subject⟩ is".
The consequence survives, in a form we can now state against the benchmark's own materials. §4.3 measured subject-late probes at 0.951 and 0.962 at layer 5 — the layer EasyEdit's shipped config edits. RippleEdits' own prompt form therefore receives about 95% of the update at the standard edit layer. The concern was never that a probe begins with the subject; that was our probes' shape, not the mechanism's requirement. The mechanism needs only a high coefficient, and at the layer the field edits, both forms have one.
What the correction forecloses is the depth half: at layer 20 that form falls to ~0.58, so a deep edit would reach these prompts far less. We were not claiming that, and now cannot.
That measurement has since been run, and §4.6 is it. On their edits and their prompts: 0.969–1.000 for the same-entity criteria, 0.982 for the compositional ones, against a floor of 0.070 built by scoring every prompt a second time against a stranger's
k*. The compositional row is the one we did not expect — their tests about a different entity still name the edited subject verbatim, so the update arrives in full at a prompt designed to probe something else.What we would like decided is now narrower and sharper: is "a benchmark's different-entity tests receive the whole update at a token they did not intend to be load-bearing" a finding the field would act on — or the expected behaviour of a subject-keyed method, interesting to nobody who already knows ROME is subject-keyed? We cannot tell from inside, and it is the difference between a result and a restatement.
And a second question we would rather have asked than answered. §4.6 ends by showing the coefficient is a property of a token rather than of a prompt, with the subject and final positions a factor of 35 apart in the same forward pass. Everything this paper measures is delivery at one token. Is a delivery measure that cannot see the attention path to the answer position worth building on — or is the honest next instrument one that measures arrival at the position the model actually reads?
-
Is n = 42 with an existence claim publishable anywhere, or does this need the frequency work it currently forbids itself?
-
Is it a contribution to show that a standard check cannot support the inference drawn from it? (He et al., 2026) reported the depth phenomenon and the cosine decay first, and read the effect as depth on the warrant that edit success is stable across layers. §4.4 shows edit success cannot measure edit strength. Matching on belief displacement instead, the effect survives. Is that worth anything on its own?
8 · What is open
Nine threads are open or active. They are listed because the useful question to a reader who knows the field is not "is this right" but "which of these is worth a month" — and we cannot rank them from inside.
Table 10. Open threads, what each would decide, and what it costs. Ordered by
what we would spend next, which is a judgement we would like contradicted.| thread | the question | what a result would settle | cost |
|---|---|---|---|
| T-086 | What sets how far an edit travels? | Seven candidates eliminated at a pre-registered floor; the best pairing of the survivors is bounded at 0.534 in closed form. Nothing explains more than ~18% alone or ~28% together. | unknown — and now searched, which is the more useful form of unknown |
| T-082 | Is GPT-J's higher floor real, or unfamiliarity? | Whether §4.3's narrowing of the floor to a Llama fact is confirmed or an artifact of running Llama-gated subjects on another model | hours |
| T-095 | Does the key still reorient when context cannot reach it? | E-041 answered T-078: attention never writes more of the divergence than the MLP at any depth (largest share 1.03), and a matched-length unrelated prefix still produces ~60% of a related one's divergence mid-network. But the MLP's input comes through attention, so only an ablation separates them. | a day |
| T-081 | Is the floor-plus-displacement structure a birthplace fact? | §4.5 is measured on one relation. Editing occupation instead inverts the prediction; a clean crossover would establish the mechanism. | a day |
| T-083 | Why is the different-subject floor lowest near the edit layer? | Both models' floors dip near depth fraction 0.15, which is roughly where ROME edits. Either a connection to ROME's own causal-tracing layer choice, or a coincidence at n≈12. | a day |
| T-054 | CounterFact runs no possession check | Its premise was reversed by D-002; it needs re-reading rather than running | reading |
Two further threads are answered but worth naming because they cost us claims:
T-085 killed an efficacy-metric critique in an hour by joining two files
already on disk — and has since been partly reversed by a third file: ROME optimises
v* until the target saturates, so post-edit log P(target) has an sd of 0.02 against
the true answer's 5.68, which makes ROME's ES margin rank-identical to the true
answer's suppression (ρ = 1.000 exactly). The thread asked what ES fails to distinguish and
answered with ES in disguise. The sharper critique — ES measures true-answer suppression
and nothing else — is open again, with ‖Δv‖ as a quantity tracking propagation as well
as ES does (−0.423 against +0.425) while nearly orthogonal to it (−0.276).
The meta-result, and the reason §5 is the shape it is. Classifying this project's own 17 results by comparison shape found every narrowing came from varying a dimension previously held fixed, and none from adding arms to a dimension already varied. The three never varied anywhere were model, edit layer, and edited relation. Two of them are now §4.3 and §4.4. The third is T-081.
Artifacts
Every record behind this paper is listed in the evidence ledger — all 33 decisions, 18 findings and 67 threads, generated from the repository so it cannot drift from it. Most of them went nowhere; that is the point of publishing the whole list rather than the ten claims this paper defends.
| evidence ledger | the complete record, generated |
agents/shared/decisions.md | 33 entries; falsification stated before each run |
agents/shared/findings.md | 18 literature entries, including the two that killed directions |
threads.md | 67 threads — 47 answered, 8 parked, 7 open, 2 active, 3 dropped |
results/*.json | per-item records, E-011 … E-018, T-075 |
logs/ | every run, levelled and committed, including the failures |
src/possession.py, src/discrimination.py | the shipped measure and the paired control |