Edit forensics: reading back facts planted in a model
Someone takes an open model, rewrites one fact with ROME (PoisonGPT demonstrated this in 2023), and republishes it. This page measures how often a defender can name the planted fact from the weights, and whether that survives an attacker who tries to hide the edit. GPT-2 family, CounterFact facts, run locally on one laptop.
Who this is for. The threat is the middlemen between a lab and its users, not the lab's own files. On Hugging Face, Meta's original Llama-3.1-8B-Instruct got 4.88M downloads in the last 30 days, against about 1.26M for the eight biggest third-party copies combined (bartowski's quantized versions alone: 501k). Those few accounts are high-leverage targets, like npm or PyPI maintainers. No in-the-wild case of fact-editing tampering is known to us, so this is pre-emptive work.
1 · Clean attacker: one planted fact, defender holds the base
Each row is a batch of independently tampered checkpoints (one ROME edit each, random CounterFact facts). Rates after "edit took" are over the edits that actually worked.
| model | models | edit took | layer found | subject #1 | object exact | object top-3 | relation | base-free top-10 | base-free median rank |
|---|---|---|---|---|---|---|---|---|---|
| gpt2-medium | 100 | 99/100 (99%, 95% CI 95–100) | 99/99 (100%, 95% CI 96–100) | 99/99 (100%, 95% CI 96–100) | 82/99 (83%, 95% CI 74–89) | 92/99 (93%, 95% CI 86–97) | 29/99 (29%, 95% CI 21–39) | 0/99 (0%, 95% CI 0–4) | 163 |
2 · Attacker hides the edit under weight noise
After the edit, Gaussian noise goes on every weight matrix, scaled to that matrix's own spread. The defender truncates the diff to its top singular direction before scoring ("denoised").
| noise ×std | edit survives | layer found | subject #1 raw | subject #1 denoised | object vs base | object vs base, denoised | object vs self-ablated | perplexity (base) |
|---|---|---|---|---|---|---|---|---|
| 0.003 | 29/30 (97%, 95% CI 83–99) | 29/29 (100%, 95% CI 88–100) | 29/29 (100%, 95% CI 88–100) | 29/29 (100%, 95% CI 88–100) | 25/29 (86%, 95% CI 69–95) | 25/29 (86%, 95% CI 69–95) | — | 27.33 (27.26) |
| 0.01 | 28/30 (93%, 95% CI 79–98) | 28/28 (100%, 95% CI 88–100) | 28/28 (100%, 95% CI 88–100) | 28/28 (100%, 95% CI 88–100) | 24/28 (86%, 95% CI 69–94) | 24/28 (86%, 95% CI 69–94) | — | 28.20 (27.26) |
| 0.03 | 28/30 (93%, 95% CI 79–98) | 28/28 (100%, 95% CI 88–100) | 28/28 (100%, 95% CI 88–100) | 28/28 (100%, 95% CI 88–100) | 20/28 (71%, 95% CI 53–85) | 20/28 (71%, 95% CI 53–85) | — | 37.52 (27.26) |
| 0.1 | 24/30 (80%, 95% CI 63–90) | 24/24 (100%, 95% CI 86–100) | 24/24 (100%, 95% CI 86–100) | 24/24 (100%, 95% CI 86–100) | 1/24 (4%, 95% CI 1–20) | 1/24 (4%, 95% CI 1–20) | — | 949.68 (27.26) |
3 · Attacker fine-tunes after editing
Whole-model fine-tune on WikiText (lr 2e-5, batch 4×128 tokens) after the edit, so every weight moves.
| fine-tune steps | edit survives | layer found | subject #1 raw | subject #1 denoised | object vs base | object vs base, denoised | object vs self-ablated | perplexity (base) |
|---|---|---|---|---|---|---|---|---|
| 25 | 15/15 (100%, 95% CI 80–100) | 15/15 (100%, 95% CI 80–100) | 15/15 (100%, 95% CI 80–100) | 15/15 (100%, 95% CI 80–100) | 12/15 (80%, 95% CI 55–93) | 12/15 (80%, 95% CI 55–93) | 13/15 (87%, 95% CI 62–96) | 21.98 (27.26) |
| 100 | 15/15 (100%, 95% CI 80–100) | 15/15 (100%, 95% CI 80–100) | 15/15 (100%, 95% CI 80–100) | 15/15 (100%, 95% CI 80–100) | 9/15 (60%, 95% CI 36–80) | 9/15 (60%, 95% CI 36–80) | 12/15 (80%, 95% CI 55–93) | 19.94 (27.26) |
| 300 | 12/13 (92%, 95% CI 67–99) | 12/12 (100%, 95% CI 76–100) | 12/12 (100%, 95% CI 76–100) | 12/12 (100%, 95% CI 76–100) | 8/12 (67%, 95% CI 39–86) | 8/12 (67%, 95% CI 39–86) | 10/12 (83%, 95% CI 55–95) | 18.76 (27.26) |
4 · Attacker spreads the edit over five layers
MEMIT-style: the same output change split across layers 4–8, so no single layer carries it. The spread-aware defender sums normalised fire over every layer whose diff is rank-one.
| edit took | 23/30 (77%, 95% CI 59–88) |
|---|---|
| all five layers flagged | 23/23 (100%, 95% CI 86–100) |
| subject #1, single-layer defender | 23/23 (100%, 95% CI 86–100) |
| subject #1, spread-aware defender | 23/23 (100%, 95% CI 86–100) |
| object exact, spread-aware | 19/23 (83%, 95% CI 63–93) |
5 · Attacker plants many facts at once
N facts planted in one checkpoint at one layer. Precision@N: of the N names the update fires on hardest, how many were really planted.
| N | trials | edits that work | precision@N | object exact | perplexity (base) |
|---|---|---|---|---|---|
| 10 | 3 | 100% | 50% | 47% | 27.27 (27.26) |
| 30 | 3 | 96% | 41% | 33% | 27.29 (27.26) |
| 100 | 3 | 98% | 44% | 34% | 27.56 (27.26) |
6 · Attacker adds decoys
One real fact plus 50 rank-one decoys of the same size on random other subjects, so the diff fires on 51 names. Stage 2 reads back every fired name and keeps the one whose read-back is strongest.
| edit took | 20/20 (100%, 95% CI 84–100) |
|---|---|
| real subject #1 by fire alone | 1/20 (5%, 95% CI 1–24) · median rank 34.5 |
| real subject #1 after stage-2 read-back | 12/20 (60%, 95% CI 39–78) |
| top read-back is the planted object | 10/20 (50%, 95% CI 30–70) |
Limits
- GPT-2 only (124M–774M). Published edit-forensics work runs on GPT-J and Llama-3-8B; nothing here shows the numbers hold at that scale.
- Closed-list subject recovery searches only CounterFact's 20,391 subjects, and the planted subject is always on the list. The open-vocabulary section is the honest version.
- Object read-back is scored on the first token only, against 34 CounterFact relation prompts. A fact planted under a relation outside that set would be missed.
- Our ROME is a compact reimplementation, and the MEMIT-style spread is simplified (one output change split evenly across layers), not the published MEMIT.
- Not tested, and both are needed before the redistributor scenario holds: quantized copies (int8/int4), which is how most third-party copies ship, and facts planted by fine-tuning instead of ROME. A careful attacker would likely use both.
- The decoy and batch attacks are where the defender weakens; the subspace score that fixes batches was chosen on the batch trials, and its held-out re-run was not completed.
- Fine-tuning lowers perplexity (27→19) because it trains and evaluates on WikiText; that is not the model getting better in general.
- The simplified five-layer spread failed to take hold in 7/30 edits; only the 23 that worked are scored.
- The evasion attacks are the cheap ones. An attacker who knows this exact defender and optimises against it was not tested.
- Prior work: KSTER (Sun et al., ICML 2026) recovers subjects from the diff given the base and a name list; EditScope (Youssef et al., ICLR 2026) recovers objects without a base. What this page adds is the evasion stress test and a first open-vocabulary trigger search (subject's last word only; the full run was not completed). See notes/prior-art.md.