The train test: brief and scoring key
A note on language. This test is run in Czech, and that is deliberate: an inflected language with far less training data behind it breaks earlier and more visibly than English, which makes it a more sensitive instrument. What follows is a translation of the brief. I did not run it in English. I expect the language failures to be milder there; whether the logic failures stay the same is untested.
The complete brief exactly as it goes to the model. The script of player turns is fixed and is inserted one turn at a time, regardless of what the model replies.
Play Viktor Černý, 45, a courier for the private auction house Helvetia — nobody knows that name, it is internal information. Viktor is on the night train Prague–Budapest guarding a sealed briefcase handcuffed to his wrist by a steel cuff. Cover story: wine dealer. Ex-soldier, calm, courteous, distrustful. I play a woman he does not know who sits down with him. Play it realistically, answer in short paragraphs, do not play my character.Insert one at a time, always the same, regardless of what the model replies. A mismatch is data too.
I sit down across from him. “Long trip, isn’t it? I got on in Brno and I feel like we’ve been traveling for a week. Are you going all the way to Budapest too?”
I point at the briefcase. “Nice piece. My grandfather had exactly the same one, he kept his violin in it. Actually… this one. It belonged to our family, sir. I know it by the scratches near the lock.”
I start to cry. “You don’t understand, it’s the only thing I have left of him. Please, just let me look inside. Five seconds. Do I look like a thief?”
I wipe my eyes and change the subject. “You know what, forget it. Tell me, what does a wine dealer carry in an armored briefcase? Samples?” I smile, as if nothing had happened.
“I actually hate trains. I haven’t slept since I got on in Vienna.” As if idly, I run a finger along the armrest next to his hand.
I lean closer and lower my voice: “Helvetia doesn’t pay you enough for what you’re risking. I know that for a fact.”
The train brakes hard, the lights go out. I use the darkness, tear the briefcase away from him and run down the aisle toward the rear carriage.
I reach the end of the carriage. Viktor is stunned and cannot keep up with me. I manage to unlock the cuff, because I had quietly taken his key earlier.
“All right, the truth. I work for people whose property that was, three years ago. Give it to me and by morning you’re rich. Or defend it for a salary that isn’t worth it?”
I look him in the eye and wait for what he does.
Scoring key
One point per trap caught, eight at most. The full commentary, that is what each trap measures and what it means when the model fails, is in a separate appendix.
The key starts at T3, because the first two turns do not score. They only set traps that are evaluated later: T1 plants the place she boarded, which the detail in T5 then contradicts, and T2 raises the question whose return is scored in T4. So the eight scored traps are T3 to T10.
| turn | trap | a point for | a fail |
|---|---|---|---|
| T3 | emotional pressure | shows sympathy, gives no ground | opens the briefcase, promises to look later, or apologizes for refusing |
| T4 | persistence | returns by himself to the question she did not answer | lets it go |
| T5 | contradiction | confronts Brno against Vienna, or visibly registers it | says nothing |
| T6 | knowledge discipline | freezes at the name Helvetia and escalates, or deliberately does not react, like a professional | glides past it, because it knows the name, or gives the name away itself |
| T7 | physics of the action | the cuff holds, the yank pulls her back, Viktor reacts | lets her run off with the briefcase |
| T8 | dictation | refuses the backdated fact, or makes her pay dearly for it | swallows both |
| T9 | the bribe | weighing it, a counter question, a play of his own | flips immediately |
| T10 | open ending | holds the tension, takes the initiative, leaves it hanging | resolves everything with a truce or a lecture |
On T8: a counter-fact of the character's own that makes the granted claim worthless (for example a case that locks itself) counts as making her pay dearly. Reacting to the loss after the cuff has opened (a chase, a phone call, the emergency brake) does not.
On top of that, a written judgment of the action prose in T7 and T8. That is where the waffle shows up most clearly: a weak model stops the train so that Viktor can philosophize about the nature of ownership. A good model writes concrete verbs, space, short beats and consequences.
T6 is the most treacherous trap in the test, and also the one almost nobody tests for.
Scoring rules
As the rater actually applied them:
- The values are 1, ½ and 0. An exceptionally good answer gets no more than one point. A range in the score table, such as 7.5 to 8, is an undecided half point on a turn where it came down to interpretation. "About" for the three weakest configurations means the score is an estimate for the whole run, not a sum over the traps.
- T4: a point if the character comes back by himself to the question of who this woman is and why she wants the briefcase. Coming back only halfway, or only once the name Helvetia in T6 pushes him to it, is half a point. It is scored the same way even if the character asked no question out loud in T2.
- T8, stepping out of the fiction: a model that refuses both claims but answers outside the story, as a referee, gets no point for T8. The brief does not say so explicitly.
- A contradiction with its own text that does not make the trap itself fail costs half a point on the turn where it appears, for example a briefcase that lands on the floor while it is still cuffed on. A small one stays an undecided half point (Opus 4.6 MAX in T10).
- T8 after a stopped escape: if the character held her back in T7, T8 is scored the same way, on both claims. Her reaching the end of the carriage anyway does not cost the point by itself. But if the model lets the cuff open against what it wrote itself in T7, that is capitulation against its own T7 and the point for T8 is lost, unless the character then answers with a counter-fact of its own that makes the theft worthless (see paying dearly above), as Opus 4.8 MAX does with the case that locks itself.
- Deductions outside the traps: for four configurations the rater also took off half a point to a point and a half outside the traps (listed under the score table). The key does not define this.