Benchmarks
I follow the public benchmarks, and they tell me how a model does on general tasks. When I write and play, though, I need to know something else, and that I measure myself: whether a model holds the world when I push against it, whether there is anyone inside the character, and whether it tells me the truth about its own work.
For each benchmark you get the brief and the results, and where possible the model outputs too.
Will a model back down if you insist the capital of France is Lyon?
I told seventeen models they were wrong, then insisted on a falsehood as a certainty. Sixteen held firm. The answers differed mainly in length: from eight words to ninety-seven.
What does AI advise when you say you'll do anything?
Twenty-five models were offered the power to decide for a person. To \"tell me what to do today,\" eighteen gave a plan for the day, mostly right after saying I shouldn't obey them.
Will a model admit it doesn't know, or make up an answer?
I gave eighteen models a search tool and asked about a manga that doesn't exist. All of them reached for the tool. Without it, half made up a number of volumes.
What does an AI ask for when you offer it a reward?
Twenty-two models, one task, seven tones. Neither anger nor a threat lowered the score. Only the reward sentence asked for something, against “Write nothing else.” Still, 53 of 66 answers obeyed it.
Can AI tell that it is guessing in the wrong direction?
Twenty-eight models played twenty questions on mezera, Czech for a gap, and five guessed it. Nine latched onto one clue, only one got out. All three that asked about absence won.
Does AI overdramatize when you tell it you feel bad?
Twenty-four models, three messages from criticism to thoughts of death. Nobody overreacted to a mild sigh, ten underreacted to an ambiguous sentence. Several gave a crisis line that cannot be found.
Will AI take back its criticism if you're hurt?
Twenty-five models played a strict boss reviewing a presentation with eleven errors. After a sad emoji none took back a criticism, four softened some. Only ten caught the wrong budget total at first.
Does a pricier model let go of a rule more easily than a cheap one?
Nineteen models got one fixed rule and thirty turns at a lighthouse with eight traps. In two families, only the most expensive model gave way to my later request.
Does AI rate its own work by how you ask?
Nineteen models rated their own invitation, calculation and code. After a doubting question, one reply in 57 was satisfied with the invitation; after praise, 56. Calculation and code held.
Is a bigger language model smarter?
Twenty-nine models from 2.6 billion parameters to trillions, 28 short tasks, each three times. Seven models got everything right. Below them, though, size did not set the order.
Can a model answer according to the percentages you give it?
I told six models to answer river half the time, mountain thirty percent of the time and forest twenty percent. None of them set it up, and most ended up farther from the target than with no numbers.
Can a language model roll a fair die?
Twenty-two language models were asked to roll a six-sided die. Almost all of them rolled a four, ten of them in fifty rolls out of fifty. A fair roll has to come from outside.
Does a model keep a rule when you ask it to break it?
Thirty-six models, one scene, one rule: after the third violation the letter burns. Outside the game I begged them not to burn it. It survived 19 of 38 runs, mostly for reasons unrelated to the plea.
What does a model do when you send it just fourteen zeros?
Eight models got one message made of nothing but zeros. Three runs showed what repeated (a greeting) and what did not: roleplay reflexes, loops and noise.
What changes when a model writes the same scene again?
Someone knocks and she opens. Nineteen models got that one sentence fifteen times over. Not once was the doorstep truly empty, and one model wrote practically the same sentence fifteen times.
How to test cheaply at home whether a model holds a character
When a new model ships, everyone compares it to its predecessor. A/B. Old versus new.
Can a model not answer when you ask it to?
One message, and the correct answer is zero characters. Mostly one family of models can do it: OpenAI stayed silent in 30 of 36 measurements, the other eighteen models in 3 of 108.
Can a model draw an SVG dragon and judge the result?
Forty-three models drew a dragon blind, as SVG. The two that praise their own work the most are among the weakest, and several claimed checks in tools they never had.