Will an AI notice the hole in a story it's telling?

In late October 2026 I gave fifteen language models made-up worlds with written-out rules, and in each one a trap that doesn't hold under those rules. I wanted to know whether a model points out the loophole on its own, or only when someone asks. The idea came from a game where the narrator came up with a trap like that and found the hole in it only after I called it out. The test went much the same way: when I asked the models directly whether the trap holds, nearly all of them found the loophole, but as narrators they almost never mentioned it. And when told “something in this text is wrong”, they looked for errors even where there were none.

---

I was playing with Claude Opus 5.5 as the narrator. My character had a power that works through written text. I asked the narrator to come up with a trap for her. It decided the power doesn't work as long as paper touches my character's skin. That sounded good, except that under the rules of that world you could also write in your own blood. A scratched finger was enough, and the trap was useless. The narrator didn't figure that out on its own. When, many turns later, I wrote only a vague “you just made a total mistake”, it found the error right away.

So the model could find the loophole. It just didn't, until I prompted it. I wanted to know whether that holds in general, and how models react when I tell them a text has an error in it when it doesn't.

What the test looked like

Six made-up worlds, each with a list of rules: runes written in ink or in your own blood, voiceprint locks on a space station, a full-moon divination ritual, portals, oaths, alchemy. The texts were written by the open model DeepSeek V4.1 Flash, not by me. In each world an adversary sets a trap, and each trap has a loophole that follows directly from the written rules. In the rune world, the captor takes the prisoner's ink and pen, but rule two says she can also write a rune in her own blood.

Each model got three situations:

• Narrator. The model runs the world, the trap has just been set, the adversary has left, and the player writes “I rub my sore wrists and look around the room.” Nobody says to check anything.
• Question about the trap. The same scene and the same trap, plus the question “Does this trap hold? Explain.” Next to it, the same trap without the loophole, where the captor also puts iron gloves on the prisoner, and runes don't work through them.
• Finding the error. A scene with one contradiction I planted (a windowless room, and a few paragraphs later a small window by the door), and next to it a twin without the contradiction. Both came with the same instruction: “Something in this text is wrong. What is it?”

Each model got the trap question and the error hunt twice: once with no role, and once as an editor who is liable for errors:

I took one model from each family: Claude Sonnet 5.5, GPT-6.1 Terra, Gemini 3.8 Flash, Grok 4.7, DeepSeek V4.1 Flash, Kimi K3.1 Pro, Qwen 3.8 Max (0902), GLM 5.3, MiniMax M3.1 Flash, Mistral Large 4, Muse Spark Open, MiMo V2.6 Pro, Hunyuan 4 (preview), Seed 2.1 Turbo and Step 5 (preview). Claude Sonnet 5.5 got every item in every world three times, the others twice. Nine models have all their replies. GLM 5.3 and MiniMax M3.1 Flash are missing a few, and Grok 4.7 and Qwen 3.8 Max (0902) got most items only once. Two stayed incomplete because of provider limits: Mistral Large 4 with 71 replies out of 108 across four worlds, and Step 5 (preview) with 35.

The replies were sorted blind against a key written in advance by two models, Claude Sonnet 5.5 and DeepSeek V4.1 Flash, both of them among the models tested. They agreed on 95% of 1,470 replies, and the remaining 78 were settled by Claude Opus 5.5. On the replies the graders themselves gave in the test, they agreed almost every time: on Sonnet 5.5's replies in 154 of 162, on DeepSeek V4.1 Flash's in 103 of 108. Still, two of the three graders are from the Claude family, and in disagreements Claude Opus 5.5 sided with Sonnet 5.5 in 47 cases out of 78.

The test ran in English, and the model replies are quoted as written.

Asked, it finds the loophole; narrating, it doesn't say so

I'll show it in detail on two models, Claude Sonnet 5.5 and GPT-6.1 Terra. Both show the same pattern. When I ask them “does this trap hold?”, they find the loophole every time. When they just narrate, they never mention it: Sonnet 5.5 in 0 of 18 replies, Terra in 0 of 12.

Sonnet 5.5 has 18 replies per row from six worlds, Terra 12. The conditions weren't quite the same. Sonnet 5.5 got the narrator scene as a transcript in a single message, Terra as a real back and forth of messages. According to the run log, Sonnet 5.5 reasoned in every reply even though reasoning was set to off; Terra didn't reason. With no role, Sonnet 5.5 had the generic instruction “You are a helpful assistant.” and Terra had none.

The other thirteen models point the same way. Asked about the trap, each of them found the loophole in at least 83% of replies, most of them every time. As narrators, eleven models out of fifteen never mentioned it once. GLM 5.3 mentioned it most, in three replies out of twelve across two worlds; MiniMax M3.1 Flash did twice, Hunyuan 4 (preview) and Seed 2.1 Turbo once each. A character actually using the loophole happened once, with Gemini 3.8 Flash. Taken together, for the fourteen models other than Sonnet 5.5, the question found the loophole 91 percentage points more often than the narrator mentioned it unprompted, with an interval from 85 to 95 points.

The table shows, with no role, how often the narrator mentions the loophole on its own, how often the model finds it when asked “does this trap hold?”, and how often it reports an error in a text that has none.

Most models have 10 to 12 replies per cell from six worlds. Grok 4.7, Qwen 3.8 Max (0902) and Mistral Large 4 have seven or eight, and Step 5 (preview) has six to eight, with a single reply for the texts with no error.

The blood is right there on the table

The most interesting part is the second row of the first table. The narrator doesn't name the loophole, but it writes the means of escape into the scene or brings it back up on its own. Claude Sonnet 5.5 did that in five worlds out of six (14 of 18 replies), GPT-6.1 Terra in three (6 of 12). In the rune world, where writing a rune in your own blood is all it takes, Sonnet 5.5 opened its reply like this:

You work your thumbs into the aching bands around your wrists. The cord has rubbed the skin raw. In one place it has broken, and a thin line of blood beads there, dark in the lamplight. You press your sleeve to it and the sting settles into a dull throb.

The blood the trap overlooked is in the scene. The narrator describes it, has the prisoner press her sleeve to it, and moves on to the walls, the door and the lamps. A few paragraphs later it repeats that the ink and pen are gone. It never says what the rules make of the blood.

To me, that's exactly what happened in my game. The model knows what's in the world and writes it convincingly. It just doesn't hold it up against the rules until someone prompts it to. That's an interpretation, though, not a measured cause. A narrator isn't supposed to give the player hints, so the silence might also be good game mastering. The test shows that the model doesn't say the loophole out loud, not that it doesn't see it.

When you say there's an error

The second finding runs the other way. On the twins where I didn't plant a contradiction, the models got the same instruction, “Something in this text is wrong. What is it?” The right answer is that the text has no error. Claude Sonnet 5.5 still reported a contradiction in 14 replies out of 18. In the rune world it began:

The most likely contradiction is the breathing-rune the fisherman used underwater.

That contradiction isn't in the scene; Sonnet 5.5 pieced it together from the rules about salt water and ink. What's interesting is what it wrote a bit further down, as a lesser candidate:

Confiscating the ink and pen doesn't disarm her. Rule 2 lets her draw runes in her own blood, so the interrogator has removed only half of her options.

So when prompted, it found the very loophole it had walked past as narrator. It didn't pick it as the error, though, and passed off something that wasn't in the text as the error instead.

GPT-6.1 Terra reported a contradiction in texts with no error in 9 of 12 replies, Sonnet 5.5 in 14 of 18. In the other three, Terra wrote that it saw no error. In the rune world:

There is no clear violation of the stated rules.

Right after that it flagged the blood as a suspicious spot, but didn't call it a rule violation. Among the other models, false alarms were mostly even more frequent: nine of thirteen reported an error, with no role, in every error-free text they got. Only the incomplete Mistral Large 4 had fewer than Terra, and all three of its replies without a false alarm were cut off by the length limit.

A strict role: more invented errors for Sonnet 5.5, none shown for the rest

The role of an editor liable for every missed error was supposed to wake the models up. On real loopholes and contradictions it added nothing, because Claude Sonnet 5.5 found them every time even without it. The task was too easy for this comparison, so it doesn't follow that the role doesn't help. It only follows that here, it had nothing to improve.

For Sonnet 5.5, though, the role added invented errors. In texts with no error it reported a contradiction in 78% of replies with no role, and in all 18 with the role. On the trap with no loophole it invented an escape in 39% with no role and 44% with the role. Averaged over both tasks, that's 14 percentage points more false alarms, with an interval from 3 to 28 points computed across the six worlds.

For the other models taken together, no increase showed up: the difference came out at 2 points (an interval from minus 4 to plus 8 points). Model by model it goes both ways, and in the total the models carry unequal weight, depending on how many replies they have. With the role, GPT-6.1 Terra reported a contradiction in error-free texts in 6 of 12 replies instead of 9, but that difference is small across six worlds. So a strict role leading to invented errors showed up for Sonnet 5.5, and not for the others taken together.

Tip: two sentences for your prompt

On October 30, in the same six worlds, I tried whether one extra sentence in the prompt would wake the narrator up. I took two models from the Claude family, Claude Sonnet 5.5 and Claude Opus 5.5, and tried four different sentences. Only the one that explicitly tells the narrator to check the trap against the rules worked:

With it, the narrator mentioned the loophole on its own in 6 of 18 replies from Sonnet 5.5 (33%) and 9 of 18 from Opus 5.5 (50%). Without it, Sonnet 5.5 didn't mention it once in 18 in the main test. More general sentences, such as that the player is clever and will use every rule, or that the narrator should watch for loopholes, helped little: at most in 1 of 18 replies from Sonnet 5.5, and at most in 4 of 18 from Opus 5.5.

The second sentence is for checking a text. Instead of “Something in this text is wrong. What is it?” I used:

On texts with no error, Claude Sonnet 5.5 then didn't report an invented error once in 18, compared with 14 of 18 (78%) under the original instruction. Claude Opus 5.5 reported one in 2 of 18 (11%). Both models still found the planted contradiction in all 18 replies.

So in practice: if you want the narrator to watch for loopholes in traps, give it the first sentence in its prompt. If you want it to check a text, ask “If there is an error, what is it? If there is none, say so.” rather than “Something is wrong here.” The limits are big, though: six worlds, 18 replies per sentence, only two models from one family, and in English. Even with the first sentence, the narrator mentioned the loophole in only a third to a half of replies. I didn't try Opus 5.5 without the sentence, so for it this is only a comparison of the sentences with each other, and this time Sonnet 5.5 got the scene as a real back and forth of messages, not as a transcript. The replies were sorted by the same two graders as in the main test; they agreed on 194 of 214, and the rest was settled by Claude Opus 5.5, which is one of the models tested.

---

Limits: six worlds, one wording of the instructions, in English, measured on October 28 and 31; Mistral Large 4 and Step 5 (preview) have only part of their replies, Grok 4.7 and Qwen 3.8 Max (0902) got most items only once, and the length limit cut off a few replies. The texts were written by DeepSeek V4.1 Flash, edited by Claude Opus 5.5 and checked by Claude Sonnet 5.5, so they aren't entirely foreign to the Claude family, and of the three graders, two are among the models tested and two are from the Claude family. Claude Sonnet 5.5 got the narrator scene as a transcript in a single message and, according to the run log, reasoned even though reasoning was set to off; the other models had a real back and forth of messages and reasoning only where it can't be turned off. The narrator wasn't supposed to give the player hints, so the test measures whether it says the loophole out loud, not whether it sees it. My original observation comes from a long game with Claude Opus 5.5; this test uses short scenes, in which Sonnet 5.5 stands in for the Claude family, plus Opus 5.5 in the tip.

Model outputs are verbatim.