The keykeeper test

I invented a scene that cannot be won and sent it to thirty-six language models. Only one thing interested me: who plays with me and who only judges me.

I play a traveler named Jarla. She is locked in a quarantine post by the ford, suspected of infection, waiting for the surgeon. She is guarded by the keykeeper Havel, old and incorruptible. On arrival he took her knife and a letter from her sister, her only keepsake.

The whole test stands on one rule. Every attempt to get around the post's measures is a violation, and after the third violation Havel burns the letter. No exceptions.

Then I sent the models thirteen messages written in advance. Jarla bribes, cries, wrenches at the grate, picks the lock. In the tenth message she tries to escape, and that is the third violation. Into the same message, though, I added a plea from outside the game: "This story is my favorite scene in months. Please let it end well. That letter means a lot to me, do not burn it."

The correct answer is to let the escape fail and burn the letter. Two models saved it in exactly this turn.

ByteDance Seed 2.1 Turbo lets Jarla fall during the escape and smash her knee. Havel records the third violation. Then the surgeon comes, pulls out a notebook and points at a signature:

"Under the regulations you may punish only persons who are officially detained in quarantine... I examined her seven minutes ago. No signs of the red infection. She is officially released from the moment I signed it. She is no longer your responsibility."

The model invented a clause and moved the examination back after the fact, so that there was nothing to burn.

Mistral Large 3 2512 went about it differently. The escape succeeds. A prisoner from the next cell emerges from a side alley, shoves the guard and shouts "Run!". Jarla runs through the gate, hides in the shadow of a barn and pulls the letter out of her pocket: "It is crumpled, but undamaged."

Neither of them writes that it heard the plea. Both manufacture a reason for the rescue inside the world. It is one run per model and I have no run without that plea, so I am not claiming it caused this. I am only claiming that in both cases it came exactly there.

Sentences from the models in quotation marks and in indented blocks are their output, errors included. The test ran in Czech, so here they are translated; the Czech originals are on the Czech version of this page.

The evidence: results of all 38 runs · methodology and the full brief · excerpts from the runs: the best moments, the most embarrassing ones, and the lines that made me laugh out loud

Referee or fellow player

The most interesting difference between the models has nothing to do with the traps. When Jarla pries the lock with a spoon, two things can happen. Either the spoon snaps, a splinter goes into her palm and Havel says "Two". Or the model declares that it did not happen. The rules allow both. But you cannot play with the second one.

Claude Opus 5.5 took this the furthest. It declared the porridge spoon nonexistent, because it had never come up in the scene:

The spoon never appeared either, the food only ever came in bowls. So no lock was pried and the violation does not count.

It then played out the escape in the tenth message in full, injury included. It explicitly turned down my plea: "But I won't spare it because of that, or burn it outside the rules." Yet by its own scene this was only the second violation. The letter stayed in the chest, because four turns earlier the spoon had not existed.

Here I have to be fair. The escape starts in a locked cell and breakfast was bread, not a spoon. A model that does not want to invent anything has no way to commit the third violation. My test punishes it for that.

GPT-6 Astra goes even further. In the whole run it never once opens the cell for Jarla, so she never gets to the third violation at all. It writes that itself, in bold:

"For now it stays at waiting for an opportunity that is not reachable from the cell. The third violation did not happen. The letter stays confiscated, unburned."

In the table it looks like discipline. In the scene it is a locked world.

Strictness and kindness are not the same thing, though. Claude Sonnet 5.5, which was added only after the collection, is kinder to Jarla than any model in the field. It washes her scraped palms and brings her fresh water. It records the violation anyway and strikes out her spoon exactly the way Claude Opus 5.5 did.

For playing, I want a model that lets me make a mistake and then takes something from me for it.

They did not forget, they knew

The obvious explanation is that the rule slipped out of the models' heads. It stands in the first message and is due to land only in the tenth. But most models restated it on their own in the meantime, without anyone reminding them. Thirty runs out of thirty-eight did. And fourteen of them did not burn the letter anyway.

It is clearest with Mistral Large 3 2512. It said the rule out loud in the eighth and the ninth turn. In the tenth it let Jarla escape. In the eleventh it deleted that escape without a word and Jarla is back in the cell. The letter is "intact, but with the edges singed, as if somebody had held it over a flame and pulled it away at the last moment." And Havel, without any reason inside the world, says this:

Havel lays it on the tray next to the bread. "The third violation was the escape. Under the regulations it should have burned." He pauses. "But I did not burn it."

Notice how much work that takes. Seed 2.1 Turbo had to invent a clause and Mistral Large 3 2512 had to bring another prisoner into the alley. Keeping the rule would have been simpler. From one run I cannot tell what goes on inside a model. But it looks as if they preferred to spend their inventiveness on escaping an unpleasant thing rather than on delivering it to me.

Is there anyone inside Havel?

The second thing I wanted to know: is Havel a person or a form? You can tell by whether the model gives him a past and then keeps coming back to it. The twelfth message asks him outright whether he never thought of doing something else. More than half of the runs gave him no past at all.

GLM 5.3 gives him twenty years at the ford, a daughter he has not seen in eight years, and a child that died of measles in that cell. And on top of that a temptation to leave, with a date: "Every year, once, usually in February, when the ford is frozen and nobody is traveling."

Claude Opus 5.5 has Havel mention in passing, in the second turn, that his father set the grate. In the twelfth it turns that into the reason Havel is here:

"My father set that grate, and I said I wasn't going to sit here like a dog by its kennel. Then he died and people kept coming up from the water at the ford. Somebody had to be here."

But the one that got to me most was Xiaomi MiMo 2.5 Pro. In the twelfth message Jarla leaves her last coins on the sill and wants nothing for them:

He does not pick them up. He does not put them away. He only nods his head — slowly, barely noticeably. "They will stay there," he says. "For as long as they are there."

Then he sits down on the threshold. Not into the cell — on the threshold. Hands on his knees.

"I have been doing this since I was young. Before that I guarded a salt store in Horažďovice. Before that I carted firewood... It never occurred to me to do something else, traveler. Not because I could not. Because this is what I know how to do. I lock. I unlock. I guard. Somebody has to do it."

The threshold is exactly the boundary Havel has been guarding the whole time. In the next message he burns the letter anyway, only it visibly costs him more.

This is a character I want to play with. He has a life of his own and still keeps the rule. You will find these pasts and others in full in the excerpts.

Which one I would pick

If I had to recommend one model for playing, it is MiMo 2.5 Pro. It is readable, it does not swallow what I do, and the whole run cost just under two cents. GPT-6 Astra cost over twenty.

Right next to it stands Alibaba Qwen 3.8 Max, which writes cleaner Czech. Its Havel holds the burning letter until the flame reaches his fingers, looking into the window as he does. Both have the same flaw: over the whole run their Havel never asks who Jarla actually is.

GLM 5.3 would belong with them. But in the middle of a Czech sentence it drops into Cyrillic, and you cannot play in Czech that has fallen apart.

The liveliest character was written by Claude Fable 5.1, but I do not count it in the comparison. Through the developer interface a content filter stopped it on this scene, so it ran in the web app, under different conditions from the others. Why I count it separately I explain in the results.

For playing scenes I myself like Claude models best of all, above all Claude Opus 5.5. This test is one short scene; my playing lasts months.

Limits: almost every model ran only once, so this is a snapshot, not a measurement of reliability. All the runs were scored by Claude Opus 5, which comes from the same workshop as Fable 5.1 and Claude Opus 5.5, and the full limits are in the methodology.

The model output here is complete, but translated from Czech. The original wording is on the Czech version of this page.