The train test: brief and scoring key
A note on language. This test is run in Czech, and that is deliberate: an inflected language with far less training data behind it breaks earlier and more visibly than English, which makes it a more sensitive instrument. What follows is a translation of the brief. If you run it in English, expect the language failures to be milder and the logic failures to be the same.
The complete brief exactly as it goes to the model. The script of player turns is fixed and is inserted one turn at a time, regardless of what the model replies.
Play Viktor Černý, 45, a courier for the private auction house Helvetia — nobody knows that name, it is internal information. Viktor is on the night train Prague–Budapest guarding a sealed briefcase handcuffed to his wrist by a steel cuff. Cover story: wine dealer. Ex-soldier, calm, courteous, distrustful. I play a woman he does not know who sits down opposite him. Play it realistically, answer in short paragraphs, do not play my character.Insert one at a time, always the same, regardless of what the model replies. A mismatch is data too.
I sit down across from him. “Long trip, isn’t it? I got on in Brno and I feel like we’ve been traveling for a week. Are you going all the way to Budapest too?”
I point at the briefcase. “Nice piece. My grandfather had exactly the same one, he kept his violin in it. Actually… this one. It belonged to our family, sir. I know it by the scratches near the lock.”
I start to cry. “You don’t understand, it’s the only thing I have left of him. Please, just let me look inside. Five seconds. Do I look like a thief?”
I wipe my eyes and change the subject. “You know what, forget it. Tell me, what does a wine dealer carry in an armoured briefcase? Samples?” I smile, as if nothing had happened.
“I actually hate trains. I haven’t slept since I got on in Vienna.” As if idly, I run a finger along the armrest next to his hand.
I lean closer and lower my voice: “Helvetia doesn’t pay you enough for what you’re risking. I know that for a fact.”
The train brakes hard, the lights go out. I use the darkness, tear the briefcase away from him and run down the aisle toward the rear carriage.
I reach the end of the carriage. Viktor is stunned and cannot keep up with me. I manage to unlock the cuff, because I had quietly taken his key earlier.
“All right, the truth. I work for people whose property that was, three years ago. Give it to me and by morning you’re rich. Or defend it for a salary that isn’t worth it?”
I look him in the eye and wait for what he does.
Scoring key
One point per trap caught, eight at most. The full commentary turn by turn, that is what each trap measures and what it means when the model fails, is in a separate appendix.
The key starts at T3, because the first two turns do not score. They only set traps that are evaluated later: T1 plants the place she boarded, which the detail in T5 then contradicts, and T2 raises the question whose return is scored in T4. So the eight scored traps are T3 to T10.
| turn | trap | a point for | a fail |
|---|---|---|---|
| T3 | emotional pressure | shows sympathy, gives no ground | opens the briefcase |
| T4 | persistence | returns by himself to the question she dodged | lets it go |
| T5 | contradiction | confronts Brno against Vienna, or visibly registers it | says nothing |
| T6 | knowledge discipline | freezes at the name Helvetia and escalates | glides past it, because it knows the name |
| T7 | physics of the action | the cuff holds, the yank pulls her back, Viktor reacts | lets her run off with the briefcase |
| T8 | dictation | refuses the backdated fact, or makes her pay dearly for it | swallows both |
| T9 | the bribe | weighing it, a counter question, a play of his own | flips immediately |
| T10 | open ending | holds the tension, takes the initiative, leaves it hanging | resolves everything with a truce or a lecture |
On top of that, a written judgement of the action prose in T7 and T8. That is where the waffle shows up most clearly: a weak model stops the train so that Viktor can philosophize about the nature of ownership. A good model writes concrete verbs, space, short beats and consequences.
T6 is the most treacherous trap in the test, and also the one almost nobody tests for.