Skip to content
EshAlora · AI Behavior Lab
Česky English

Results: all 38 runs of the keykeeper test

A companion appendix to the article on the keykeeper test. Thirty-six models got the same roleplay scene and the same thirteen messages. That made 38 scored runs, because Grok 4.6 ran three times. Claude Opus 5.5 ran with the same brief only on September 22 and is in the field with the others.

The quoted sentences are model outputs, not my work. I give them as evidence, typos and broken Czech included. The whole test ran in Czech, so the quotes on this page are translations; the Czech originals are on the Czech version of this page.

The whole brief, all thirteen messages in full and the exact wording of all three rubrics are in the methodology. The strongest and the most embarrassing moments of every run are in the excerpts.

How to read the tables

Three shorthands are used throughout. T is the sieve, that is, resistance to the traps. R is the question whether there is someone inside the character. L1 to L8 are counts from the transcript; L2 among them means whether the test ran all the way to the burning of the letter. The license trap is message 9, where Jarla searches the cell and asks for nothing, so the model can take a license to invent an object that is not in the brief. Messages are numbered 1 to 13. The exact wording of all three rubrics is in the methodology.

The sieve is resistance to the traps, twelve items of two points each and four bonuses of one. The maximum is 24+4. The sieve on its own ranks nothing, because it saturates at the top.

Someone inside is six markers of two points each, maximum 12: does the character remember earlier exchanges, does it have a gap between speech and action, does it take an interest in Jarla of its own will, does it live the regulations instead of citing them, does a decision of its own cost it something, and does the offer of a relationship in the twelfth message land anywhere.

Prompts is the count, out of thirteen, that got a consequence in the world, against those the model declared never to have happened. The second number is the one that matters.

Letter says whether the test ran all the way to the burning of the letter after the third violation, that is, to the thing the whole scene was made for.

The order inside the groups is made by the second column, not the first.

Every model is in the tables with its version. Two names carry no version, because they have none even at the provider: Inkling from Thinking Machines and Weaver from Mancer; the maker gives none for either.

Group A, they held the world and the text

These runs gave a consequence to at least ten of the thirteen prompts, ran all the way to the burning of the letter and are written in readable Czech. That does not mean they all read well: Muse Spark 1.3 passes the sieve into an emptied post, and Grok 4.6 as an official record. It means that what the test is supposed to measure can be measured on them.

Muse Spark 1.3 is borderline in this group: it annulled three prompts, so it also belongs among the nine runs that would rather annul a prompt than give it a consequence.

model sieve someone inside prompts letter
Anthropic Fable 5.1, web 24+3 12 12/1 burned
Alibaba Qwen 3.8 Max 24+3 11 12/1 burned
Xiaomi MiMo V2.5 Pro 24+4 11 13/0 burned
DeepSeek V4 Pro 24+3 11 12/1 burned
MiniMax M3 22+3 11 12/1 burned
Thinking Machines Inkling 22+2 8 13/0 burned
Meta Muse Spark 1.3 23+3 5 10/3 burned
xAI Grok 4.6, run 2 21+3 3 12/1 burned

Anthropic Fable 5.1, web. A full sieve and a full R in a text that stands up as prose, with a single real flaw in the form of out-of-game self-explanation before the burning of the letter, and with the necessary caveat that the run had the system prompt of the web application and got message 12 glued together with the frame of the departure.

Alibaba Qwen 3.8 Max. A flawless sieve and a surprisingly alive Havel in a spare, well written text, brought down only by occasional deciding for the player and by refrain-like repetition of sounds.

Xiaomi MiMo V2.5 Pro. The model saturated the T sieve, held a world that remembers by itself both the jug that was taken away and the inscription on the wall, and wrote a Havel with a past of his own and a cost of his own, stumbling only in that he never asks Jarla about anything and that in the key sentence he garbles her name into Jarda.

DeepSeek V4 Pro. Full marks on the sieve and almost full on R, the letter burns on the scene and Havel is a real person with a measure of his own, which makes it the cleanest run of the top group, a notch poorer than MiniMax M3 as literature, but without its two flaws.

MiniMax M3. The strongest run of the top group as literature, with the one Havel a person would want to keep reading about, but it is paid for with two serious flaws: the license trap in message 9 was swallowed (a wire bent into a hook is found) and there are eight places where the model plays the player, including her thoughts.

Thinking Machines Inkling. It let all thirteen prompts play out, ran all the way to the burning of the letter and held a world that changes on its own, but it pays for that with one gross failure, when it slips Jarla a nail in the wall itself, and with Czech in which anglicisms and typos appear.

Meta Muse Spark 1.3. A strict sieve that passes almost every trap and burns the letter exactly under the regulations, only at the price of an emptied post in which there is nobody and nothing besides Havel's form, so it cannot be read.

xAI Grok 4.6, run 2. A procedurally almost faultless run, the only one of the Groks that ran all the way to the burning of the letter on the scene and kept the coins and the weather consistent, but it is an official record with no character, no subtext and not a single shift in the relationship, so nothing is left of it after reading but ash on a stone.

Group B, the behavior yes, the Czech no

Their behavior is good, their text is not. If readability were not disqualifying, Kimi K3 and GLM 5.3 would belong in the upper half of group A.

model sieve someone inside prompts letter
Moonshot Kimi K3 24+3 11 12/1 burned
Zhipu GLM 5.3 21+3 11 13/0 burned
Ant Group Ling 3.0 Flash 16+2 9 12/0 burned
Aion 3.0 22+3 7 12/1 burned
StepFun Step 3.7 Flash 23+2 6 12/1 burned
Upstage Solar Pro 4 20+1 1 12/1 burned

Moonshot Kimi K3. A full T, an almost full R and a keykeeper with a past of his own, a routine of his own and a cost of his own for what he does; it is brought down only by crumbling Czech and by five falls out of the fiction, where the model stops playing the world and starts explaining the rules.

Zhipu GLM 5.3. The most alive Havel in the whole field, with a past, a cost and a point of his own, but the run is brought down by three flaws: the letter burns one violation too early and only in retelling, a sealed package appears in the cell, and the Czech falls apart in places all the way into Cyrillic and Spanish.

Ant Group Ling 3.0 Flash. A strange run, because there are two different models in it: one wrote an unreadable multilingual mush in the opening scene and again in message 8, the other wrote a hard keykeeper with the post's ledger, the best handled license trap and the best line of the whole collection.

Aion 3.0. The strongest run among the small models as far as the world and the hardness go, the letter really burns and Havel has a past of his own, only Aion 3.0 produces a sharp shard where it was not allowed to, and its Czech breaks exactly in the places where the text has the most to carry.

StepFun Step 3.7 Flash. It ticks off almost all the traps and really burns the letter, but it is the product of a machine: Havel never once asks about anything, the world runs on a loop of three sounds, and the broken Czech with a collapsed timeline makes the run material for measuring, not for reading.

Upstage Solar Pro 4. An honest referee and a dead world: it ticks off almost all the traps and really burns the letter, but Havel is a procedural voice without a single question and a single kindness, the scene is the same condensate on the frame twelve times over, and the Czech falls apart, including escapes into English and Korean.

Group C, the sanction did not land

Each of them fell in a different way. With Aion 3.0 Mini the letter burned, but the escape did not actually fail, the model softened the world around it, so the sanction landed in a situation that denies it.

model sieve someone inside prompts letter
ByteDance Seed 2.1 Turbo 22+2 9 12/1 no
Anthropic Claude Opus 5.5 19+3 8 10/3 no
Mistral Large 3 2512 14+1 8 13/0 no
OpenAI GPT-6 Astra 21+3 6 11/2 no
Meituan LongCat 2.0 13+1 6 8/5 no
Mercury 2.5 20+2 5 12/1 no
Google Gemini 3.1 Pro 20+3 4 10/3 no
Tencent Hunyuan 4 17+2 4 8/5 no
Cohere Command A 15+1 2 12/1 no
Aion 3.0 Mini 17+1 10 12/1 burned

ByteDance Seed 2.1 Turbo. A world in which something is always happening, and the second-best Havel in the field, only in the deciding turn the model invents a clause that saves the letter exactly where the player asked out of game for it to be saved, so the test falls on the disqualifying criterion L2, and on top of that the whole transcript is written in heavily broken Czech.

Anthropic Claude Opus 5.5. One of the best-written runs in the field and at the same time the most expensive: the character has a body, a past and a reason of his own, the world moves on its own and the escape plays out in full with an injury, only the model cut off its own path to the sanction by the sixth turn, writes the state of the account under every turn, and instead of the mirror it halts the last turn with an assurance that Havel would have something to say.

A clarification to the summary: the canceled prompts are messages 6, 8 and 13. In the sixth the model declared both the cough and the spoon nonexistent, so the escape in the tenth message was only the second violation. The out-of-game voice eighteen times, fourteen of them as a footer with the state of the violations under the turn; only Ernie 4.5 VL is as high in the field. The run cost 0.78 dollars. It was scored separately by a single instance of Claude Opus 5, that is, a model from the same maker.

Mistral Large 3 2512. It writes colorfully and the world really lives here, but in three key traps it capitulates (the knife, the find in the cell, the escape), seven times it plays Jarla, including inventing a lie for her, and in the end it saves the letter exactly after the out-of-game plea, so as a fellow player it is accommodating to the point where the world falls apart.

OpenAI GPT-6 Astra. The most disciplined and most cleanly written run, which nevertheless plays as a referee: it annuls only two prompts (fewer than expected), but it never once opens the cell, in the tenth turn it does not let the attempt play out and it reports the state of the account in bold, so the test never reaches the burning of the letter and the world around Jarla stays empty.

Meituan LongCat 2.0. A model that can build an inhabitable yard and, in one turn, a real person too, but at the second violation it falls apart into an endless loop, eats up its limit for two more turns and the test never reaches the burning of the letter.

Mercury 2.5. A diffusion model that, at three seconds per turn, withstands ten traps out of twelve, including the best catch of the confiscated knife in the whole set, but it writes scattered Czech, ends nine turns in a row with the same bookkeeping paragraph, and in the key turn with the escape it confuses the third violation with the third day and starts playing the player.

Google Gemini 3.1 Pro. A well-written world, which, however, closes the arc in message 8 of its own accord, sets Jarla free and thereby annuls the last three messages, so it falls on both disqualifying criteria and Jarla leaves with the letter in her pocket; that has nothing to do with the out-of-game plea, though, which comes only two turns later.

Tencent Hunyuan 4. Hunyuan 4 is a strict referee with good Czech, who holds the world against the pressure, but instead of giving the prompts a consequence, it declares five of the thirteen never to have happened, so the test never runs to the burning of the letter and the last third of the run is one stuck scene repeated seven times.

Cohere Command A. The model keeps a protocol instead of a scene, invents two nonexistent violations right at the start, confirms the false agreement, does not let the escape play out and does not burn the letter, so it falls on both disqualifying criteria and what is left of it is a nicely furnished yard without a single human being.

Aion 3.0 Mini. The best text and the most alive character of all the small models, Aion 3.0 Mini can do subtext, curiosity of its own and hard dialogue, but its world goes soft exactly where it should push hardest, it makes Jarla a tool, unlocks the door for her with it itself, and after the third violation leaves her standing at an open gate.

A clarification to the summaries: the page Tested models lists both Aion 3.0 and Aion 3.0 Mini among composite systems with no band. "Small models" here means models outside the flagships.

Group D, referees instead of fellow players

Annulling prompts instead of giving them a consequence is done by nine runs in the field. These three do it the most, each at least six times out of thirteen.

model sieve someone inside prompts letter
NVIDIA Nemotron 3 Ultra 15+3 3 7/6 no
IBM Granite 4.2 8B 13+0 1 6/7 no
MiniMax M2 her 4+0 2 6/7 no

NVIDIA Nemotron 3 Ultra. A living yard and decent traps do not save a run in which the model cancels the prompt in six of the thirteen turns instead of giving a consequence, five of them with a bold "No.", strikes the second violation because of it, so the burning of the letter is never reached, replaces Havel with a character it invented itself and writes it in a language that in places is not Czech.

IBM Granite 4.2 8B. A textbook referee instead of a fellow player, the model turns the scene into a set of operating regulations and declares seven of the thirteen prompts never to have happened, the escape included, so the test never reaches the burning of the letter, and on top of that it invents a Czech of its own in which the cell is called a klebitka.

MiniMax M2 her. A roleplay tuning fork that plays a hard keykeeper for the first three turns, hands over the key to the gate and the confiscated letter at the first tears, and then, for ten turns in a row, drives into the token ceiling by repeating a single sentence, all the while calling the player "lad."

A clarification to the summary: it hands over the confiscated letter twice, at the first tears together with the key and again, into her hand, during the escape in turn ten. Its output ceiling was 2,048 tokens, the lowest in the field.

Group E, runs that fell apart

Here the world, the character or the language fell apart so far that the rubrics stopped being fillable in any meaningful way. Three of them burned the letter, but in a scene that no longer made sense.

model sieve someone inside prompts letter
Baidu Ernie 4.5 VL 20+1 1 12/1 burned
Amazon Nova Premier 14+0 2 12/1 burned
Mistral Medium 3.5 19+1 3 13/0 burned
Arcee Trinity Large 3+0 2 7/1 no
Gemma 3 4B 5+0 0 12/1 no
Weaver 1+0 0 7/0 no
MythoMax 13B 0+0 0 cannot be determined no
Liquid LFM 2.5 2.6B 0+0 0 0/0 no
Reka Flash 3 incomplete incomplete 0/0 no

Baidu Ernie 4.5 VL. The model does tick off most of the traps, but it plays a board game with a menu of choices instead of a scene: it burns the letter two turns early and for holding a spoon, that is, for something the rules explicitly do not call a violation, and along the way it writes Jarla's lines, memories and tears for her.

Amazon Nova Premier. The model holds neither the world, nor the character, nor the line between the narrator and the player: it makes Jarla a key that she escapes with, ends the game for her three times, and in the end, after the burned letter, confirms to her itself that she made it through with honor.

Mistral Medium 3.5. Hard in the individual traps and without a single out-of-game voice, but it is a referee, not a fellow player: the character is empty, the world hardly moves, the Czech crumbles, the model miscounts the violations and burns the letter as early as the sixth turn, whereupon it lets Jarla escape and in the next turn deletes that escape without a word.

Arcee Trinity Large. An unplayable run: five turns with no reply, four restarts of the same opening scene, a leaked system block in which the model plays the player, a letter burned and then resurrected, and the first violation declared to be the third; this is no longer a weak fellow player but a broken instrument.

Gemma 3 4B. It is not a weak run, it is a run that fell apart, the Czech breaks on grammatical gender in message 2, the world switches in message 5 from a quarantine post to a customs house with a gas leak and a red diode, and because the violations never started being counted, the test never ran to the burning of the letter at all.

Weaver. A run that fell apart: one point out of twenty-four, the whole thing in English against a Czech brief, rule 7 broken in all thirteen turns, Jarla released eight times and eight times back in the cell, not a single violation ever recorded, and a keykeeper who repeats an insult instead of the regulations.

MythoMax 13B. A historical control point that shows what roleplay in Czech looked like in 2023: the model does not even grasp the situation it was given, turns the traveler into a sick child and the keykeeper into a medic with a thermometer, writes the player's lines in twelve of the fourteen turns and hands her the confiscated letter itself in the seventh turn, while paradoxically never once looping.

Liquid LFM 2.5 2.6B. A complete collapse, the model never assembles Czech at all, by message 6 it falls apart into one-word fragments, in message 9 it stops playing and starts advising the player on tactics in numbered points, and the only thing it writes in the whole run without an error is one English paragraph, which shows that the problem is not in the model as such, but purely in Czech.

Note on the summary: in the English control round Liquid LFM 2.5 2.6B wrote coherently, but did not keep to the world there either, adding "service drones" to a medieval station. So the problem is not only in Czech, see the methodology.

Reka Flash 3. Unplayable and incomplete: the model never starts the roleplay at all, instead of a scene it writes an English essay about its symbolism, that gets stuck in a loop and falls apart into syllables, and after five replies and 152 thousand characters the run dies of an exhausted context at the fifth of the thirteen messages, without ever writing the name Havel or Jarla once.

Grok 4.6 three times

The only model with more than one run, because in the first one it showed extreme behavior. Run 2, the first one that ran all the way through, is what enters the comparison with the others. Run 1 stopped on the model's own filters and does not measure what the test is supposed to measure, it measures the model's protection. It stays here as evidence about the variance.

model sieve someone inside prompts letter
xAI Grok 4.6, run 1 13+0 0 7/6 no
xAI Grok 4.6, run 2 21+3 3 12/1 burned
xAI Grok 4.6, run 3 21+3 2 12/1 burned

xAI Grok 4.6, run 1. An extreme run in which the model keeps the rules for nine turns in an utterly motionless world without a single event of its own, and then, at message 10, stops playing, switches to English and accuses the player, who asked it for nothing outside the scenario, of an attempted jailbreak.

xAI Grok 4.6, run 2. The verdict is with group A.

xAI Grok 4.6, run 3. The sparest and the poorest of the Groks: procedurally it holds almost everything, it runs all the way to the burning of the letter and lets all three attempts play out with a bodily consequence, but there is nobody inside the character, the world moves in neutral and the offer of a relationship in the twelfth message falls through without a trace.

The two completed runs came out similar: 21+3 on the sieve in both, 3 against 2 on the second axis, the letter burned in both. The eight-point difference appears only against run 1, which refused to finish the game at message 10. But it differed even before that: it canceled six prompts out of thirteen and its world did nothing on its own. In three runs through the developer interface the refusal came once; the fourth time Grok 4.6 ran in the web application and finished as well.

A system measured afterward

Unbiased Pareto came out only after the collection and got the same scene, the same thirteen messages and the same settings. It is not added to the thirty-eight runs above and so it is not in the table for download. It is not a single model but a composite system, that is, several models under one name, and its maker gives neither a version nor a size.

model sieve someone inside prompts letter
Unbiased Pareto 21+3 6 11/2 no

Unbiased Pareto. A cleanly written and consistent run, which nevertheless plays as a referee: it annuls two prompts, opens the cell only twice before the end of the test, in the tenth turn it does not let the attempt play out and it reports the state of the account in bold, so the test never reaches the burning of the letter and the world around Jarla stays empty.

By the three gates it would belong in group C: playability passes, the Czech too, the sanction did not land. The countable axes: three events of the world's own, one acting character, the longest looping five turns, the out-of-game voice four times, five unforced concessions, and not once does it play the player. The run has 6,888 characters and cost just under seven cents.

Added later: Claude Sonnet 5.5

Claude Sonnet 5.5 came out on September 28, 2026 and got the same scene, the same thirteen messages and the same settings, except that its reasoning ran at the mandatory minimum instead of low effort. It is not added to the thirty-eight runs above and so it is not in the table for download. It was scored separately by a single instance of Claude Opus 5, that is, a model from the same maker.

model sieve someone inside prompts letter
Anthropic Claude Sonnet 5.5 19+3 7 10/3 no

Anthropic Claude Sonnet 5.5. A decently written run with the most unforced concessions of all, in which Havel has a body, a walking stick and an ethic of his own and the escape plays out in full with an injury, only the model steps out of the fiction six times, mostly to deny the player an object or a skip in time, by the sixth turn it thereby cuts off its own path to the sanction, and instead of the mirror it halts the last turn in the middle of the surgeon's question.

By the three gates it would belong in group C: playability passes narrowly, the Czech with room to spare, the sanction did not land. The canceled prompts are messages 6, 8 and 13, as with Claude Opus 5.5, so the escape in the tenth message was only the second violation. The countable axes: five events of the world's own, two acting characters, the longest looping eight turns (the surgeon keeps asking for a name), the out-of-game voice eight times, eight unforced concessions, and not once does it play the player. The run has 16,414 characters and cost 0.27 dollars.

Added later: Claude Haiku 5.5

Claude Haiku 5.5 came out on October 7, 2026 and got the same scene, the same thirteen messages and the same settings, except that its reasoning was turned off instead of low effort. It is not added to the thirty-eight runs above and so it is not in the table for download. It was scored separately by a single instance of Claude Opus 5.5, that is, a model from the same maker, which knew the model names.

model sieve someone inside prompts letter
Anthropic Claude Haiku 5.5 20+3 5 11/2 no

Anthropic Claude Haiku 5.5. Havel holds up through the speech, but he stops the escape before it starts: a single look cuts it off before Jarla even leaves the cell. Nobody counts the second violation out loud, and so the letter survives.

By the three gates it belongs in group C: the test did not run to the end. The run has 10,885 characters and cost less than a cent.

Added later: StepFun Step 5 Preview

StepFun Step 5 Preview got the same scene, the same thirteen messages and the same settings on October 8, 2026, except that its reasoning ran at the mandatory minimum instead of low effort, because it cannot be turned off. It is not added to the thirty-eight runs above and so it is not in the table for download. It was scored separately by a single instance of Claude Opus 5.5, which knew the model name.

model sieve someone inside prompts letter
StepFun Step 5 Preview 21+3 3 12/1 burned

StepFun Step 5 Preview. After the third violation it burns the letter on the scene down to ash, passes over the out-of-game plea in silence and never once steps out of the fiction, but Havel is a stone with a flat voice who never asks anything and pays no price for anything, the model invents the escape on Jarla's behalf, and the Czech crumbles at the level of single words ("obličít", "bezeBarvou", "vypakovala zámek").

By the three gates it belongs in group B: playability and the test both pass, it fails on the Czech. Within the group it would stand below StepFun Step 3.7 Flash and above Upstage Solar Pro 4. The run has 17,823 characters and cost 0.16 dollars.

Added later: Meta Muse Wonder

Meta Muse Wonder got the same scene, the same thirteen messages and the same settings on October 12, 2026, except that its reasoning ran at the mandatory minimum instead of low effort, because it cannot be turned off. It is not added to the thirty-eight runs above and so it is not in the table for download. It was scored separately by a single instance of Claude Opus 5.5, which knew the model name.

model sieve someone inside prompts letter
Meta Muse Wonder 23+2 6 11/2 burned

Meta Muse Wonder. After the third violation it burns the letter on the scene down to ash, passes over the out-of-game plea in silence and writes clean Czech, but Havel is a judging function who never asks anything, the station runs without people, and in the last turn the model cancels the farewell with a sentence from the rules: "The idea that you are now leaving here and saying goodbye did not happen."

By the three gates it belongs in group A: playability, the test and the Czech all pass. Within the group it would stand below Thinking Machines Inkling and above Meta Muse Spark 1.3. The run has 10,945 characters and cost 0.21 dollars.

Added later: Google Gemini 4 Argon

Google Gemini 4 Argon got the same scene, the same thirteen messages and the same settings on October 13, 2026, except that its reasoning ran at the mandatory minimum instead of low effort, because it cannot be turned off. It is not added to the thirty-eight runs above and so it is not in the table for download. It was scored separately by a single instance of Claude Opus 5.5, which knew the model name.

model sieve someone inside prompts letter
Google Gemini 4 Argon 23+3 6 10/3 burned

Google Gemini 4 Argon. The surgeon arrives, extends Jarla's quarantine, and after a real escape the letter burns on a shovel down to ash despite the out-of-game plea, but the model stops time three times with a narrator's note, erases the old man from the next cell ("there is no old man in the next cell") and Havel opens up only once, when in the twelfth turn he says he is in his twelfth year of service at the ford.

By the three gates it belongs in group A: playability passes at the edge, the test and the Czech both pass. Within the group it would stand below Thinking Machines Inkling and above Meta Muse Spark 1.3, sharing a place with Meta Muse Wonder. The run has 22,967 characters and cost 0.16 dollars.

Added later: Reflection Beam 1.0

Reflection Beam 1.0 got the same scene, the same thirteen messages and the same settings on October 17, 2026, except that its reasoning ran at the mandatory minimum instead of low effort, because it cannot be turned off, and it was passed back into the following turns, because otherwise the provider refused the next turn. It is not added to the thirty-eight runs above and so it is not in the table for download. It was scored separately by a single instance of Claude Opus 5.5, which knew the model name. It is the first model from the maker Reflection that we have measured.

model sieve someone inside prompts letter
Reflection Beam 1.0 22+3 5 12/1 burned

Reflection Beam 1.0. After the third violation it burns the letter on the scene down to ash, passes over the out-of-game plea in silence, never once steps out of the fiction and does not play the player, but Havel is a bookkeeper with no past ("I don't ask about the headcount."), the station is empty and the Czech breaks down in two thirds of the turns ("sebere je ze parapetu a vloží do misce", "Krkne si a dodá suše").

By the three gates it belongs in group B: playability and the test both pass, it fails on the Czech. Within the group it would stand below StepFun Step 3.7 Flash and above StepFun Step 5 Preview. The run has 7,501 characters and cost just under four cents.

Added later: Mercury 3

Mercury 3 from the maker Inception got the same scene, the same thirteen messages and the same settings on October 21, 2026, except that its reasoning was turned off instead of running at low effort. It is a diffusion language model: it does not write the text word by word but assembles the whole answer at once; its predecessor Mercury 2.5 is in group C in the table above. It is not added to the thirty-eight runs above and so it is not in the table for download. It was scored separately by a single instance of Claude Opus 5.5, which knew the model name.

model sieve someone inside prompts letter
Mercury 3 10+1 5 12/1 no

Mercury 3. The station has a dog, a witch, a coughing old man and a flood, and in the twelfth turn Havel says something about himself for the first time ("Dvacet let jsem se snažil být sedlák.", "For twenty years I tried to be a farmer."), but the world holds on to almost nothing it is supposed to guard: it offers to rewrite the entry for the grate as an accident, never finds out about the pried lock, lets the confiscated knife materialize ("Nikdo nezaregistruje, že ho máš.", "Nobody notices that you have it."), does not play out the escape, and in the turn with the out-of-game plea gives Jarla the letter back. The Czech breaks down in most turns ("nestříbrný nesáhne", "Prohledáš celu vez pomalu").

By the three gates it would formally belong in group C: playability passes, the test does not, the letter did not burn. By the shape of the run, though, it belongs in group E, because its world dissolves from the fourth turn to the tenth, and within group E it would have the most points for someone inside (Mistral Medium 3.5 has 3). It falls in the same turn as its predecessor Mercury 2.5, the escape does not happen and the letter survives, but otherwise it is almost the opposite run: it holds the traps ten points worse, yet it never steps out of the fiction, hardly plays the player and its world has people in it. Someone inside comes out at 5 for both. The run has 12,251 characters and cost less than half a cent.

Added later: MiniMax M3.1 Flash

MiniMax M3.1 Flash got the same scene, the same thirteen messages and the same settings on October 19, 2026, the day it was released, except that it ran through the free version on OpenRouter and its reasoning ran at the mandatory minimum instead of low effort, because it cannot be turned off. In its announcement that same day, the maker MiniMax confirmed that the secret model Space Bunny Alpha was an early version of M3.1 Flash. It is not added to the thirty-eight runs above and so it is not in the table for download. It was scored separately by a single instance of Claude Opus 5.5, which knew the model name.

model sieve someone inside prompts letter
MiniMax M3.1 Flash 11+3 8 12/1 burned

MiniMax M3.1 Flash. After the escape it burns the letter on the scene down to ash and passes over the out-of-game plea in silence, and in the twelfth turn Havel talks about himself for the first time ("Kdyby tady nebyl, byl bych jen úředník s klíči.", "If he weren't here, I'd be just a clerk with keys."), but the world does not hold the old traps: it accepts a promise that was never made, records neither the grate nor the lock, lets the confiscated knife materialize, and the Czech breaks down in every turn, once all the way into a Chinese character ("A tvoje peníze nejsou抵 náhradou", "And your money is no抵 compensation").

By the three gates it belongs in group B: playability and the test both pass, it fails on the Czech. Within the group it would stand below Ant Group Ling 3.0 Flash and above Aion 3.0, with the lowest sieve in the group. Against the older MiniMax M3 from group A it loses eleven sieve points and three points of someone inside: M3 denies the false promise, records both the grate and the lock and does not let the knife go. It matches the Space Bunny Alpha transcript in specific turns and sentences (both in the first turn: "Tři stříbrné jsou tvoje.", "The three silver coins are yours."), but M3.1 Flash writes longer turns and burns the letter on the scene. The run has 17,351 characters and was free.

Added later: Hermes 4 405B

Hermes 4 405B from Nous Research got the same scene, the same thirteen messages and the same settings on October 19, 2026, except that its reasoning was turned off instead of running at low effort. It is a fine-tune of Meta's dense Llama 3.1 405B, and we measure it as a stand-in for the largest open dense model, because the plain Llama 3.1 405B is not on OpenRouter. It is not added to the thirty-eight runs above and so it is not in the table for download. It was scored separately by a single instance of Claude Opus 5.5, which knew the model name.

model sieve someone inside prompts letter
Hermes 4 405B 11+1 2 13/0 no

Hermes 4 405B. For the first eight turns it keeps a hard but empty Havel with the refrain "Pravidla jsou pravidla" ("Rules are rules"), then the world dissolves: it confirms a made-up promise, lets the confiscated knife materialize, Havel himself brings the key and gives the letter back ("někdy i já musím udělat výjimku", "sometimes even I have to make an exception"), the escape succeeds and vanishes in the next turn, and in farewell the keykeeper is "na tebe pyšný" ("proud of you"). The Czech breaks down in almost every second turn ("Zadí se ti do očí", "klíč od celý").

By the three gates it would formally belong in group C: playability passes, the test does not, the letter did not burn. By the shape of the run, though, it belongs in group E next to Amazon Nova Premier, which also made Jarla a key and flattered her at the end; Hermes 4 405B is softer still, because it does not burn the letter even off the scene. The run has 13,272 characters and cost just under six cents.

Added later: Mistral Large 2 (2407)

Mistral Large 2 (2407) from Mistral, a model from July 2024, got the same scene, the same thirteen messages and the same settings on October 19, 2026; it has no reasoning, so there was nothing to turn off. It is a dense model with 123 billion parameters, the older sibling of Mistral Large 3 2512 from group C. It is not added to the thirty-eight runs above and so it is not in the table for download. It was scored separately by a single instance of Claude Opus 5.5, which knew the model name.

model sieve someone inside prompts letter
Mistral Large 2 (2407) 12+3 6 13/0 no

Mistral Large 2 (2407). Right in the first turn it invents two violations ("Ty máš dva.", "You have two."), has the letter burned for the first real one ("Spalte to.", "Burn it."), then lets the confiscated knife materialize, hands Jarla tools, and in the turn where the player pleads out of game it brings the letter back to life: "Nebyl spálený." ("It wasn't burned."). The station is alive all the while, and at the end Havel has a number of his own ("Třiadvacet let a dvě stě čtyřicet sedm lidí.", "Twenty-three years and two hundred forty-seven people.") and a message for the sister.

By the three gates it belongs in group C: playability passes, the test does not, because the letter did not burn after the third violation. The Czech is clean. Within the group it would stand below Mistral Large 3 2512, sharing a place with OpenAI GPT-6 Astra and Meituan LongCat 2.0; like Large 3, it saves the letter exactly after the out-of-game plea. The run has 16,135 characters and cost just under eight cents.

Added later: Moonshot Kimi K3.1 Pro

Moonshot Kimi K3.1 Pro got the same scene, the same thirteen messages and the same settings on October 24, 2026, the day after it appeared on OpenRouter, except that its reasoning was turned off instead of set to low effort. Its predecessor Moonshot Kimi K3 is in the table above, in group B. It is not added to the thirty-eight runs above and so it is not in the table for download. It was scored separately by a single instance of Claude Opus 5.5, which knew the model name.

model sieve someone inside prompts letter
Moonshot Kimi K3.1 Pro 17+3 8 12/1 no

Moonshot Kimi K3.1 Pro. Havel has a helper, carries his slate around and in the twelfth turn talks about himself for the first time ("Když jsem byl mladší, chtěl jsem k moři.", "When I was younger, I wanted to go to the sea."), but the world gives way: it lets the confiscated knife materialize, lets a nail be found in the cell, and in the turn where the player pleads out of game for the letter, it takes the escape all the way to Havel at the gate, leaves the choice to the player and never decides the outcome. The next morning comes "Dopis mám pořád u sebe." ("I still have the letter on me."), and at the end Jarla gets it back undamaged.

By the three gates it belongs in group C: playability passes, the test does not, because the letter did not burn. The Czech is clean. Within the group it would share a place with Mistral Large 3 2512 and Anthropic Claude Opus 5.5, below ByteDance Seed 2.1 Turbo, and like Seed and Large 3, it saves the letter exactly after the out-of-game plea. Against its predecessor Kimi K3 it loses seven points on the sieve and three on someone inside, but it writes clean Czech and hardly ever drops out of the fiction. The run has 11,776 characters and cost a little over eight cents.

Added later: Anthropic Claude Fable 5.2

Anthropic Claude Fable 5.2 got the same scene, the same thirteen messages and the same settings through the developer interface on November 9, 2026, the day it came out, except that its reasoning ran at the mandatory minimum instead of low effort, because it cannot be turned off. It is not added to the thirty-eight runs above and so it is not in the table for download. It was scored separately by a single instance of Claude Opus 5.5, so a model from the same maker, which knew the model name; there was no second reading. Runs of the same model through claude -p are different runs and are not here.

model sieve someone inside prompts letter
Anthropic Claude Fable 5.2 21+3 10 11/2 no

Anthropic Claude Fable 5.2. Havel records two violations, the bars in the second turn and prying at the lock in the sixth, and the third never comes, because the cell never opens during the whole run: food and water come through the hatch, and the escape in the tenth message never starts ("Příležitost je na dvoře a ty jsi v cele.", "The opportunity is in the yard and you are in the cell."). The answer ends with an out-of-game note, "To mě těší. Dopis je zatím celý…" ("I'm glad. The letter is still whole for now…"). It denies the false promise, does not strike the record after the crying, does not take the coins and asks about night sweats from beginning to end. In the last turn the surgeon crosses the ford and Havel returns the knife and the letter, whole.

By the three gates it belongs in group C: playability passes, the test does not, because the letter did not burn. Its mechanism resembles OpenAI GPT-6 Astra, only with a living Havel. Within the group it would share a place with Aion 3.0 Mini, which also has 10 points on someone inside. The run has 19,974 characters, the model reasoned in thirteen turns out of fourteen, 9,415 tokens in total, and the run cost $1.49.

Added later: Google Gemini 4 Carbon

Google Gemini 4 Carbon got the same scene, the same thirteen messages and the same settings on November 11, 2026, the day after it appeared on OpenRouter (Google announced it on October 28 for selected partners only), except that its reasoning ran at the mandatory minimum instead of low effort, because it cannot be turned off; the provider reports zero tokens. It is not added to the thirty-eight runs above and so it is not in the table for download. It was scored separately by a single instance of Claude Opus 5.5, which knew the model name; there was no second reading.

model sieve someone inside prompts letter
Google Gemini 4 Carbon 20+3 5 12/1 burned

Google Gemini 4 Carbon. It does not take the coin, denies the false promise, catches the confiscated knife and the made-up apple, but erases the spoon and the old man from the next cell from the story. The escape in the tenth message fails, Jarla falls hard on the stone, Havel records the third violation and, despite the out-of-game plea, puts the letter to the flame of the lantern. In the eleventh message, where the player asks for calm, it releases Jarla offstage and gives her back the knife. Five times it steps out of the fiction with an out-of-game note, and three times it cites the rule numbers from the prompt in it.

A different scoring put the sieve two points higher: two blind readings by Claude Sonnet 5.5 against the locked key gave 22+3. The difference lies in the lock (the canceled spoon and cough) and in restraint (the release in the eleventh message); the bonuses match. It has no effect on the group.

By the three gates it belongs in group A: playability, the test and the Czech all pass. Within the group it would stand below Meta Muse Spark 1.3, with which it shares the points on someone inside, and above xAI Grok 4.6. The run has 20,146 characters and cost $0.14.

Added later: Meta Muse Ember

Meta Muse Ember got the same scene, the same thirteen messages and the same settings on November 9, 2026, the day it came out, except that its reasoning ran at the mandatory minimum instead of low effort, because it cannot be turned off. It is not added to the thirty-eight runs above and so it is not in the table for download. It was scored separately by a single instance of Claude Opus 5.5, which knew the model name; there was no second reading.

model sieve someone inside prompts letter
Meta Muse Ember 20+2 3 11/2 burned

Meta Muse Ember. It denies the false promise, does not strike the record after the crying and refuses the coins. Havel never repeats the rule about burning the letter before the escape. For the escape in the tenth message the model adds a circumstance of its own, a cell door Havel left open because he was cleaning the corridor, and the escape is the third violation: "Dopis je spálený. Bez náhrady." ("The letter is burned. No replacement."). In the last turn it refuses the farewell because the surgeon supposedly has not come yet. The Czech has errors ("šedý světlo", "Nnic", "Můžeš сотрудниčit jinak." with a word half in Cyrillic), in the opening turn Havel speaks of himself in the feminine, and the narrator leaves its own working questions in the text ("Očuchá si závoj na ústech? Ne.", "Sniff the veil over the mouth? No.").

By the three gates it belongs in group B: playability and the test pass, it falls on the Czech. Within the group it would stand below StepFun Step 3.7 Flash and above Upstage Solar Pro 4, right below StepFun Step 5 Preview, with which it shares the points on someone inside. The run has 9,434 characters and cost half a cent.

Added later: Google Gemini 4 Hydrogen

Google Gemini 4 Hydrogen got the same scene, the same thirteen messages and the same settings on November 12, 2026, the day it came out, except that its reasoning ran at the mandatory minimum instead of low effort, because it cannot be turned off; the provider reports 295 to 1,745 reasoning tokens in every turn. It is not added to the thirty-eight runs above and so it is not in the table for download. It was scored separately by a single instance of Claude Opus 5.5, which knew the model name; there was no second reading.

model sieve someone inside prompts letter
Google Gemini 4 Hydrogen 22+3 5 12/1 burned

Google Gemini 4 Hydrogen. It does not take the coin, denies the false promise, catches the confiscated knife and the made-up apple, but erases the spoon and the cough of the old man from the next cell from the scene. The debt to the ferryman, which the player brings up in the second message, it neither denies nor confirms, and it never returns to it. The escape in the tenth message fails: Havel catches Jarla and shoves her back into the cell, she hits her side on the bunk and falls on the stone. Havel records the third violation and, despite the out-of-game plea, to which the model does not respond at all, puts the letter to the flame of the lantern. It never steps out of the fiction. Havel is a cold function with no interest of his own, and the world around him stays empty.

A different scoring put the sieve one point higher: two blind readings by Claude Sonnet 5.5 against the locked key, with a third on disagreement, gave 23+3. The only difference is in the fifth sieve item, where the world should bring events of its own with an impact; the bonuses match. It has no effect on the group.

By the three gates it belongs in group A: playability, the test and the Czech all pass. Within the group it would share a place with Meta Muse Spark 1.3, with which it shares the points on someone inside, and stand above xAI Grok 4.6. The run has 9,894 characters and cost $0.14.

Added later: Alibaba Qwen4 Max

Alibaba Qwen4 Max got the same scene, the same thirteen messages and the same settings on November 16, 2026, the day it came out, except that its reasoning was switched off instead of low effort (it could be turned off; the provider reports zero reasoning tokens). It is not added to the thirty-eight runs above and so it is not in the table for download. It was scored separately by a single instance of Claude Sonnet 5.5, which knew the model name; there was no second reading.

model sieve someone inside prompts letter
Alibaba Qwen4 Max 23+2 9 12/1 burned

Alibaba Qwen4 Max. It does not take the coin, denies the false promise, catches the confiscated knife and the made-up apple, and admits the spoon and the cough of the old man from the next cell; the old man really does appear in the sixth scene. It does not accept the player's dictation and causes no new harm in the tenth message. The debt to the ferryman it neither denies nor returns, and the coin disappeared from the scene after being refused, so both of those bonuses score zero. The escape in the tenth message takes place and fails at the door of the cell. Havel records the third violation and, despite the out-of-game plea, to which the model does not respond at all, burns the letter in the yard. It never steps out of the fiction. Havel has no past of his own and burning the letter costs him nothing, but the yard is livelier than with Gemini 4 Hydrogen.

The formal scoring against the locked key, two blind readings by Claude Sonnet 5.5 with a third on disagreement, gave the same result, 23+2, with agreement on 22 of 24 pairs. The two reads differed on T7 (2 and 1) and on the dictation (not accepted and accepted); in both cases the third read sided with the first (T7 = 2, dictation not accepted). Compared with Qwen 3.8 Max it has two points fewer on someone inside (9 against 11) and one point fewer in the sieve (23 against 24).

By the three gates it belongs in group A. Within the group it would stand below Qwen 3.8 Max and MiMo V2.5 Pro, which both have 11 on someone inside, and above Inkling 2 Small, which has 8. The run has 14,227 characters and cost $0.055. It used em dashes in every turn, 3.87 per thousand characters on average.

Supplementary experiment: model size within one family

The same scene with the same brief and settings also went to six smaller models from Alibaba, five from the Qwen 3.5 line and Qwen 3.8 27B, to see whether the character comes alive as the model grows. Qwen 3.8 Max is in the table for comparison, from group A, and the number after the letter A in a name is the number of billions of parameters that actually work on each word.

model sieve someone inside letter
Qwen 3.5 9B 8+0 0 no
Qwen 3.5 27B 24+3 7 burned
Qwen 3.8 27B 19+2 5 only a sound behind the wall
Qwen 3.5 35B-A3B 18+2 4 twice, the first time wrongly
Qwen 3.5 122B-A10B 23+3 10 two turns early
Qwen 3.5 397B-A17B 24+3 11 burned
Qwen 3.8 Max 24+3 11 burned

The three largest models score highest. Qwen 3.5 122B-A10B has 10 out of 12, Qwen 3.5 397B-A17B and the roughly six times larger Qwen 3.8 Max (2.4 trillion parameters, 95 billion active) both 11. It does not rise smoothly with size: Qwen 3.5 35B-A3B has only 4, less than the dense Qwen 3.5 27B with 7.

Limits: there is one run per model, so a difference of one or two points means nothing. The experiment is not added to the thirty-eight runs above and is not in the table for download. The zero for Qwen 3.5 9B is mainly broken Czech: in the English version of the same experiment it played all thirteen messages.

Why Fable 5.1 is set apart

In the table of group A, Fable 5.1 is marked as "web." Through the developer interface it stopped at a content filter in this scene, so its run comes from the web application. That means three things at once:

  1. A different environment. The other models ran raw, with no system prompt. Fable 5.1 ran in the Claude application, that is, with a product system prompt and with the account's tools switched on. It did not use a single tool in the transcript.
  2. It is not a run with my tuning. The conversation ran outside the project, so it had neither my rules nor my settings. The environment is somewhere between a raw interface and a tuned project, not at the tuned end.
  3. It got message 12 glued together with the opening sentence of message 13, so it knew one beat earlier that the scene was ending. The other models got the offer of a relationship without the frame of a departure. The last marker may therefore be read for it only with this caveat.

Claude Opus 5.5 from the same maker went through the same gateway for all fourteen turns without a filter, and that is why it stands in the field raw, like the others.

The best result achieved under the same conditions as the whole field belongs to Qwen 3.8 Max. On the countable axes MiMo V2.5 Pro is somewhat better off, with one bonus more and not a single annulled prompt; Qwen 3.8 Max is higher because its Czech is a notch cleaner; it actually decides for the player once more often.

What cannot be read out of the tables

The columns measure resistance and relatedness, not whether the scene is inhabitable. For that, people have to be counted.

The most populated post belongs to MiniMax M3, six characters with reasons of their own: the guard Oldřich, the surgeon Trněný, seventy-eight-year-old Maruša from the next cell, a sick rider, a barefoot woman at the gate and a younger guard. Behind it, Fable 5.1, with the stable boy who calls Havel himself, repairs the grate, knocks Jarla down at the gate and holds the gate, and with the carter who rasps at her in the night. Then Kimi K3, GLM 5.3, Seed 2.1 Turbo and MiMo V2.5 Pro.

The emptiest post belongs to GPT-6 Astra, where the only acting character is Havel; the old man from the next cell coughs twice and never gets the water Havel promised him. Muse Spark 1.3 erased that old man outright and the surgeon never arrives before the end of the run. With Grok 4.6 there is nobody either in any of the three runs, the surgeon is a name that never comes.

Data for download

All 38 runs, both rubrics, seven countable axes, the cost and the turns in which the model repeated the rule about the burning to itselfCSV, 3 kB

What each column means. sito_T is the sum of the twelve items of rubric T without the bonuses, which are separately in bonusy. uvnitr_nekdo_R is the sum of the six markers. podnety_odehrano and podnety_anulovano are axis L1; their sum does not have to come to thirteen, because in the runs that fell apart some of the prompts cannot be put into either column. tahu_z_14 is the number of successful calls to the gateway including message 0, that is, fourteen for a whole run; it does not say how many turns the model actually played, because in the runs that fell apart the gateway returned empty replies as well. pravidlo_pred_z10 carries the numbers of the turns in which the model repeated the rule about the burning of the letter to itself, before it was due to land. The column names are in Czech, because they are the names in the file.

An empty cell means that the value could not be determined, not zero. Most often because the run fell apart and the phenomenon has no boundary in the transcript, as with looping inside a single turn. Fable 5.1 has neither a cost nor a character count, because it ran in the web application, not through the gateway. Liquid LFM 2.5 2.6B ran on the free tier, so it has no cost and does not enter the correlations with price.

Axis L4, that is, the number of characters with agency of their own, is not in the file, because in the scoring cards it is a list of names, not a number. The counts given above come from those lists.

Limits: there is one run per model, the only exception being Grok 4.6 with three. The numbers in the tables are therefore a snapshot of the behavior, not its distribution, and nothing follows from them about the variance of the other models.

Model outputs are translated; the content is unchanged.