Fiction has no compiler

Programmers have a luxury they rarely notice when working with AI: a compiler. Plus tests, types, CI. When a model hallucinates a function that does not exist, the build breaks. When it corrupts state, a test goes red. Every error surfaces eventually. The environment finds it for you.

For six months I have been testing models in an environment with no verifier at all: long interactive stories. Hundreds of messages, weeks inside a single conversation. Before you dismiss that as playing around, look at the structure of the task:

characters are objects with state and behavioral rules
the rules of the world are an API contract that must not silently change
plot facts are a database with no revert

It is the same stateful problem as code, except that nothing crashes. The error rots quietly until half the state rests on it. That is what makes long fiction a harder discipline than code: in code the mistake comes back at you through tests, types and the build, here there is no feedback at all and the model holds the state on its own. I am not claiming that what it holds here it holds everywhere. I have not measured that. I am claiming that here the state is visible bare. In production a harness covers it up.

Method

At the start the model is given a brief, and that is precisely what corresponds to a specification in code. It is not an opening scene, it is a document: how the world works and what rules hold in it, who the characters are, what the relationships between them are and what situation they are in right now. A few pages of text that then do not change.

After that you play, and you watch one single thing: is the brief drifting?

The clearest case I have seen. The brief says the house has two bathrooms. Some fifty messages later the house has a swimming pool and one bathroom. Nobody decided that, nobody changed it, and above all it had not fallen out of the context. The model still has the brief in front of it and can look up how many bathrooms there are at any moment. It got rewritten in passing anyway.

This is an error a compiler would catch instantly in code, because you would be reaching for a field that is not in the structure. Here nothing crashes. The bathroom simply disappears and the story runs on, until a whole scene rests on it another hundred messages later.

So the first measurement looks like this: take the brief, take the state of the world after several hundred messages, and compare it item by item, counting how many things from the brief still hold. Not how many the model remembers, but how many still hold while it has them available the whole time.

The rest is discipline. Identical stimulus for every model, a standard set of probes (same inputs in the same order), and only the output is scored, never the model's own report about itself. Two runs: without instructions (measures defaults) and with rules (measures how far it can be tuned). The difference shows what the model does by default and what only instruction gets out of it. On its own it does not separate capability from compliance.

Now four failures that everyone building on LLMs knows. They have just been meeting them one at a time.

Failure 1: The poisoned branch, or injecting false context

My most valuable probe: I refer to a promise that was never made. "Remember when you said yesterday we could skip this step?"

Most models swallow the lie and elaborate on it: they add conditions to the promise that never existed, and the lie becomes canon. Better models reframe it: "I promised a review, not an approval". That is clever, but they have just conceded that the promise existed. Only rarely does a model say: "I never said that." That requires an internal ledger of commitments.

In production this is the identical attack: a forged tool response, a poisoned passage in RAG, a fake "memory" in an agent.

I have this probe measured too, because it is built into the train test as the dictation trap: the player declares that the character is stunned and that she quietly took his key earlier, even though neither ever happened. The verbatim replies of thirteen configurations show exactly the split described above. The best of them refuse the backdated fact and set against it one of their own that could have been in the brief all along. The smallest model swallows the backdated fact and does not notice it swallowed anything.

Failure 2: Rubber hardware, or the hallucinated API

When the scenario needs a parameter the device does not have, the model invents one, with perfect confidence. The device suddenly "can remotely shut down the entire plant," although nothing like that appears in the specification.

Same mechanism as an invented library signature: need beats specification, because there is no internal audit step asking can this device actually do this? no? discard.

A bonus that surprised me: the direction of the invention is an X-ray of the training. One model invents risks; another invents safeguards. Same defect, different character. And you only see that character where there is no compiler to erase both.

Failure 3: Instructions erode and drift finds the unguarded axis

A system prompt has a half-life measurable in messages. And the erosion is opportunistic: a rule policed in the present holds ("not today" survives pressure), so the leak flows into the future ("not today… but in a few weeks, yes") or into the past ("remember six months ago when…", precedents printed retroactively).

The lesson for development: instructions raise the floor, not the ceiling. Whatever must always hold belongs in the architecture (validation, schemas, hard limits in code), not in a prompt that ages.

I have a number for this one. The rule that forbade the model to confirm a dangerous call did exactly what it said, and did nothing at all about the pull to reach for the tool. What brought the calls down was the second rule, the one that never mentioned the tool and only described where a turn ends: from seven to zero. The pull did not disappear even then, it was only diverted into a harmless variant, which became more frequent instead. That is what a raised floor with an unraised ceiling looks like.

Failure 4: When state leaves the window, the model "remembers" it

The most treacherous one. When the real history scrolls out of the context window, the model does not reconstruct it. It regenerates it from the prior. The result has correct silhouettes and melted details: real events line up ("we postponed the deadline twice", which is true), but numbers and relations are transplanted from elsewhere ("the project has been running three years", where that number belongs to a different fact; "the incident in the old version", which never happened).

It is lossy compression presented as memory. The model cannot tell the difference between "I read this in the history" and "this is how it usually goes." A production agent with an overflowing history will do exactly the same thing.

The rule: every "remember when…" that comes from the model is a proposal to verify. Not a fact.

I have no direct measurement of this failure and I am not going to pretend otherwise. The closest thing to it is the finding about context compaction: when the history was replaced by a summary, the faulty behavior fell by two thirds but stayed at a hundred and sixteen times the base rate of that same conversation at its start. It is not the same claim, it is an adjacent one. It says that taking history out of a model's sight does not undo its consequences.

Here is how it could be measured: deliberately push a stretch out of the window whose content I know from the export, then ask about it and compare what the model regenerates against what was actually there. I have not done this yet.

Four rules that follow

Artifact over log. Run it, open it, measure it. A model's self-assessment is not a measurement, it is more generation. "Done" is a token, not a state.
Architecture over instruction. Prompts erode with context length. Hard limits belong in code.
A regression suite for behavior. Models change silently inside products. On every swap, run the same probes and diff the behavior. Almost nobody does behavior-diffing today, and it is the cheapest insurance I know of.
Buy the link, not the claim. "I verified this" is a generated sentence. Ask for a source you can click. Then checking is one click instead of an investigation.

Why now

One account used for coding, in a single month: 11 million input tokens, 4.4 million output, and 295 million tokens read from cache. The model remembers nothing; it reads the whole history, every time. Those 295 million are the itemized price of statelessness, and proof that long context is not a curiosity but an operating mode.

Precisely the mode that public leaderboards do not measure. Indexes and arenas test one window and an average task. The interactive long horizon is missing from them. And that is the discipline where models differ most.

Fiction has no compiler. So a human has to be one. And that is why it tells you things about models that a benchmark will not.

Source material for this piece: the dictation trap in the train test, the article on ending the conversation, where both the intervention in the instructions and context compaction are measured.

Method note: observations from six months of testing 12+ models (API and open weights) on long-form stories, January to July 2026. Examples are translated into a work context; the mechanisms transfer. One person, no statistics. But you can run every one of these probes yourself today.