weights · context · provenance · revision
how language models represent, update, and fail to revise what they know
When you give an LM new information that contradicts its weights, it does not detect a contradiction and decide to update. The attention mechanism simply weights the context window more heavily than the prior encoded in weights — because it was trained to do so. What looks like a decision to update is continuous attention-weighted integration. There is no internal event that corresponds to "I am revising my belief."
A system operating only in two modes cannot be corrected persistently. Tell it today that a belief is wrong: it incorporates the correction locally, this session. Tomorrow it has forgotten. For models deployed at scale — where different users, at different times, can identify wrong beliefs — there is no mechanism for those corrections to accumulate and persist. The model is incorrigible in the structural sense.
Train a neural network sequentially on Task A, then Task B. Performance on Task A degrades sharply — often to near-zero. The weights that encoded Task A are overwritten by the gradient updates for Task B. The brain does not do this. The complementary systems prevent it: the hippocampus encodes new experiences without touching the neocortex directly; slow consolidation during sleep integrates the new with the old incrementally, interleaving replays of old and new memories.
LMs have weights (neocortex-like: slow, structural) and context (hippocampus-like: fast, recent). But the two are not connected by any consolidation mechanism. Context never becomes weights. Weights never become episodic. There is no sleep loop. The result: a system where new episodes are perpetually transient — they cannot become stable knowledge without retraining, which risks the catastrophic forgetting the brain naturally avoids.
"The hippocampal system allows for the rapid acquisition of new information, whereas the neocortical system is a slow learning system that draws on regularities across many experiences... The hippocampus rapidly binds together the cortical representations active during an experience, and gradually instills this information in the cortex through a process of consolidation."
This paper — Complementary Learning Systems — is still the foundational reference. The problem it identified in biological systems is precisely the problem that LM architectures have not solved.
Provenance is the ability to trace a belief back to its evidence. Not just to hold the belief, but to know why it is held — what observations, what reasoning, what prior beliefs produced it. A provenance-aware system can answer: "I believe X because of Y, which arrived at time T, which superseded my prior belief Z which was based on W." This chain is what makes revision meaningful rather than arbitrary.
When context overrides weights, the model has no internal event that registers "I am revising." The output reflects the new information, but the prior belief is not archived — it is simply outweighed. There is no chain. The model cannot tell you why it believes what it believes, nor that it used to believe something different. It lives, as it were, in a perpetual present — each output assembled fresh, with no memory of the states that preceded it.
When a person learns that a prior belief was wrong, they do not simply begin outputting the correct answer. They hold the correction, the prior belief, and the evidence for each simultaneously — and can reason about the tension between them. They know they changed their mind. They can explain why. They can weight the new evidence against the old and remain uncertain if the evidence warrants it. None of this is possible in a system with no provenance.
Published forty years ago in classical AI — before neural networks dominated — this framework formalized what it would mean for a rational agent to revise its beliefs correctly. The three axioms are minimal: any weaker and the revision becomes meaningless; any stronger and you add constraints the problem doesn't require. LMs were not designed with these axioms in mind, and satisfy none of them reliably. The framework is a useful diagnostic: ask of any update mechanism — does it succeed, preserve consistency, minimize change?
ROME (Rank-One Model Editing) targets specific MLP layers that function as key-value stores for factual associations. It edits a single fact surgically, with minimal collateral change to other knowledge. It satisfies success (the new fact takes hold) and approximates minimal change (the edit is local). But it has no consistency mechanism — it cannot detect that the new fact contradicts other beliefs — and no provenance — the model does not know it was edited. It is the closest current approach to AGM-compliant revision, and it still falls short.
Memory-augmented architectures — Neural Turing Machines, Differentiable Neural Computers — add external writable memory. They can read and write non-destructively. But they don't scale to the complexity of natural language beliefs, and provenance is not tracked.
Retrieval-augmented generation (RAG) — pull relevant documents at inference time. The belief store is the document corpus. But revision means updating the corpus, and contradiction detection is absent — the model does not notice when retrieved documents conflict.
Lifelong learning research — EWC, progressive networks, replay buffers — each tries to prevent catastrophic forgetting. None provides provenance or contradiction detection.
The metacognitive piece: a system that knows it is revising, knows why, and can reason about the revision itself. Not just a system that outputs the correct answer after a belief update, but one that can say: "I changed this belief, it was based on evidence X, it now conflicts with stored belief Y, I resolved the conflict by doing Z." This is what human episodic memory supports. It is far beyond what any current LM architecture provides — and it may require something structurally different from the transformer: a system where belief states are first-class objects, not implicit in attention patterns over frozen weights.
The Theory of Mind problem — representing what a character believes when that differs from ground truth — is the in-context, session-local version of this problem. The model must maintain separate belief states for separate agents simultaneously. The lookback mechanism shows it can do this partially, within a story, using attention. But the deeper question is: can a model maintain persistent, updatable, provenance-tracked belief states about itself — about its own prior outputs, its own reasoning chains, its own revisions? That is what would make it a genuine epistemic agent rather than a very capable pattern-completer.
Currently OIDs encode positional identity: first/second character, first/second object. They have no time dimension. Extending to (character_OID, object_OID, belief_t) → state makes conflict detection structurally possible: two entries sharing (character_OID, object_OID) but differing in belief_t and state are a detectable contradiction. The consolidation step can then apply revision (newer wins) or merging (negotiate). The paper already uses RoPE for positional encoding — RoPE can be extended to carry belief_t alongside token position.
Sally's belief state: (Sally, marble, basket) at t=1. After Anne moves it: (world, marble, box) at t=1. After Anne tells Sally it's in the garden: (Sally, marble, garden) at t=2. An AGM-compliant lookback retrieves the active entry for (Sally, marble) — garden — when asked Sally's belief, while preserving the chain. It can answer "what did Sally originally believe?" (basket) and "what does Sally currently believe?" (garden) from the same structure, without either erasing the other.
The paper proves the read half exists and is mechanistically real — specific attention heads at specific layers perform OID-matched retrieval of (character, object, state) triples. That is the necessary foundation. AGM compliance requires the write half: a persistent, writable belief store indexed by temporal OIDs, with a consistency check on write and a non-destructive supersession chain. The lookback mechanism is not sufficient for AGM compliance, but it is the right structural starting point — the addressability and retrieval logic carry over. What must be designed is everything that happens when the world changes after the initial story is read.
Current knowledge systems accumulate claims. This one revises them. The difference is what makes it useful to researchers.
Where the consensus is solid — high confidence, no contestation, long citation history. Safe to build on.
Where it is fragile — high centrality, active debate, recent counter-evidence. Building on this is a risk worth knowing.
Where synthesis is overdue — long-running debates with a detectable middle ground that nobody has published.
Where nobody is working but someone should be — frontier adjacency gaps: methods that exist, problems that are open, connection unmade.
The graph already has 111 papers, 736 typed edges in the GNN domain. The evaluation: pick the spectral vs. spatial debate. Show that the system surfaces the contested claims with provenance chains, computes the middle ground via the merging operation, identifies which papers are in the middle ground and under-cited, and detects the frontier gap.
Compare to what a researcher finds in four hours of manual review. The system should surface the same picture in under 60 seconds — plus the gaps the manual review misses.
"I used this to find something I wouldn't have found otherwise."
Not a chatbot that summarizes papers. Not a search engine that retrieves them. A system that maps what the field does not know, and why, and where the next move is. The engineering is 70% there — claim nodes, typed edges, provenance, debates, temporal dimension. What AGM compliance adds is the semantics: supersession, consolidation, and the four gap analysis queries. Those are four new graph queries and one schema extension. The foundation is built. The contribution is the layer that makes it reason.
Formally: K*P is the set of formulae whose models are the minimal models of P under ≤K. The models of K are precisely the minimal models of the whole ordering — the worlds currently considered most likely.
Geometrically: Rank all possible worlds from most to least plausible. When P arrives, look only at worlds where P is true. Pick the top one. That is the revised belief set.
Operationally: Move as little as possible from where you are to a place where P is true. The most plausible P-world is the minimum move.
SUCCESS — you picked a P-world, so P is in K*P. ✓
CONSISTENCY — you picked a real world. ✓ (Unless P is self-contradictory — then no P-world exists.)
MINIMAL CHANGE — you picked the closest P-world. Everything not required to change, didn't. ✓
VACUITY — if P doesn't conflict with K, the K-worlds are already P-worlds. The minimum P-world is just K itself. ✓
The eight postulates are eight algebraic consequences of one geometric idea: revision respects a plausibility ordering.
The lookback mechanism retrieves states by (character, object) key. If two states exist for the same key — marble in basket at t=1, marble in garden at t=2 — there is no ordering over which is more plausible. There is only recency. The mechanism picks the most recent binding, which approximates plausibility but cannot represent source reliability, prior probability, or supersession chains.
Temporal OIDs are the first step toward a real ≤K: recency as a proxy for plausibility. The knowledge graph's supersession chains and source reliability scores are the rest. Together they give the mechanism what AGM requires — not just a retrieval key, but an ordering over what to believe when bindings conflict.