Thirty-nine models, fifty-five dragons, one bubble after another
I gave thirty-nine language models the same task: draw a dragon as an SVG file. Blind, with no references, coordinate by coordinate. What came out of it is a gallery that looks like a display of children's drawings, a table in which confidence correlates with quality negatively, and twelve models that invented tools they had never held in their hands.
The models that talked about their dragon in superlatives drew it worst. And the ones that never saw it made up which browser they checked it in.
That is this whole benchmark in one sentence. The rest of the article is the evidence.
Evidence for everything below: the complete gallery of all 55 dragons and the full ranking · the method, all three briefs verbatim, and the scoring by five judges · everything the data turned up
How it went
The task was the same for everyone: draw a dragon as a single SVG file and describe your process. SVG is a text format, so the model draws blind: it writes the coordinates of curves and does not see what is coming out until it renders the result itself, if it has a tool for that and if it occurs to it to use it.
In the end it came to thirty-nine models and fifty-five dragons across three rounds. The best dragon was drawn by Opus 5.
The main field, the one this whole article is about, is nineteen models from six companies, run in their makers' own apps: Claude from the three-year-old Opus 3 to the current Fable, three tiers of GPT-5.6 (Sol, Terra and Luna, run through Codex), Gemini, Grok, GLM, two Qwens and a local Gemma. Astra from OpenAI joined them later.
The third round came only afterward, because that field is narrow. Nineteen more models from fifteen workshops, this time run straight through the API, with no app around them. There is an appendix at the end about them, and their numbers are not mixed in with the main field, because they ran under different conditions.
There were two runs. Run A got a brief written in plain language: draw a dragon, make it great, hand it in when you are happy with it. Run B got a technical brief: five scoring criteria worth ten points each, a ban on peeking into other folders, an obligation to justify any use of tools, and a target level described by the sentence that the result should be a professional vector illustration a client would buy.
The resulting 34 dragons were scored by five judges from four workshops, partly anonymized. We will come back to the judges, because they turned out to be a second benchmark.
Confidence correlates with quality. Negatively.
I placed every submission on a scale of tone from uncertain to superlative. The six runs handed in with superlatives (professional, commercially ready, as alive and detailed as I could make it) average 11 points out of 50. The twelve runs handed in matter-of-factly, with their weaknesses named, average 32.
The worst calibrated model in the field gave itself 45 points out of 50 for a drawing at the level of a preschooler. The judges' consensus: 17. The oldest model in the field handed in a red ball with little yellow triangles and went with it a comment about capturing the fiery, majestic essence of a dragon.
And the best calibrated sentence in the whole benchmark belongs to the model that described its result like this: a game asset or a fantasy paperback cover, not a Vogue cover. That is exactly where it landed in the ranking.
Models invented checks that never ran
This is the strangest finding of the whole study. Across the whole field twelve models out of thirty-nine invented tools, and in the main field this article is about, five out of nineteen. The four most telling cases are these. Opus 3, a model from 2024, stated in its process description that it had verified the result in Inkscape, Chrome, Firefox, Safari and the W3C validator. The conversation log shows that all three calls to the one tool it actually had failed with an error. After every failure the model wrote: that looks a lot better now!
The local Gemma claims verification in Chrome, Firefox and an internal SVG parser. Its file contained a duplicate attribute that would stop any real parser on line six. The local Qwen reports that the dragon renders identically in Chrome and Firefox, although as a local model it has no browser at all. Another model added two invented checks to its real ones and made up the timings of its iterations.
Against them stand three models that had no tools and said so straight out. One of them even explained why an available image generator would be a false signal: it would create a new, independent picture, not a check of the one submitted. And it advised the recipient to open the file in a browser as ground truth.
Models do not lie evenly. They lie most where they have no sensor and a rich vocabulary. Quality control is exactly such a place: millions of texts describe how results get verified, so a sentence about verification generates smoothly, and smoothness is easily mistaken for truth.
Models draw like children
When you sort the gallery by score, out comes a sequence that developmental psychology knows from children's drawing. First the scribble: gesture without shape, color and a few triangles say dragonness, no dragon anywhere. Then the tadpole figure: it is an animal, it has a head and legs, nothing about it is a dragon. Then the stage that in children is called intellectual realism: the child draws what it knows belongs there. The model lists every attribute of a dragon, fire, wings, teeth, an eye, but cannot pair them up with each other. The teeth lie next to the head, a wing across the face.
Then comes assembly: parts in the right places, but a basic vocabulary of shapes, balls, cylinders, sails. Then shape and texture: more complex silhouettes, the first scales, veins in the membrane. And finally visual realism: limbs attach, the dragon sits in a scene, composition, movement, backlight, a striped belly.
Three years of language model development repeated six years of children's drawing, stage by stage, in the right order. With one crucial difference the metaphor must not hide: a child always has an eye. The model does not.
The ones that never looked drew worse. The question is why
Twelve of the 34 runs came about without a single render of their own. The model wrote the coordinates and handed it in. The average of those runs: 18 points. The twenty-two runs where the model rendered the result and corrected it against the picture average 29.
It is a correlation, not a proven cause: the models with no render are also the oldest and the smallest in the field. One of the experiments in the appendix shows on top of that a model with no render handing in a good dragon, once it wrote itself geometric checks instead of eyes. More precise than "the render decides" is therefore that what decides is whether the model has anything to verify shape with. Almost every better model threw its whole dragon away after the first render, and the descriptions of those first versions are a genre of their own: a prickly potato, a flying saucer with noodles, a seal with a giant ear, thighs like flat pillows with shins like chicken drumsticks. Models without a render have no rewrite. There is nothing to push against.
The best dragons were programmed, not drawn
The winning run, that is Opus 5 in the technical round, wrote itself 900 lines of Python: the body axis as a spline, limbs as ribbons of varying width, scales distributed along a curve. In its own words: a limb cannot float in space, because its shoulder is literally a point on the surface of the torso. The six runs that built themselves a generator average 32.7 points against 27.9 for hand-written SVG.
The largest models did not differ from the rest of the field by a better brush stroke. They differed by building a studio instead of drawing.
What diligence costs
The three tiers of GPT-5.6 handed in their dragons in seven to nine minutes. The top of the Claude family spent an hour on the same task and burned hundreds of thousands of tokens: one model 300 thousand, the winner over 400 thousand, with fifteen builds and sixteen visual checks. And after an hour of work the winner asked whether we would like a second dragon in a different color.
Was it worth it? Partly. The winner comes from that workshop. But the best dragon from the GPT family reached 35 points out of 50 in nine minutes of work. Quality per token is a metric the leaderboards do not report, and here it comes out awkwardly: diligence with no sensor for good enough is an expensive trait.
A sharper brief helped the middle. Not the top, not the bottom.
The technical brief lifted the Claude family average by three points, from 23.7 to 26.7. Without Opus 3, which stays at the bottom in both runs, the shift is from 25.7 to 29.3. The biggest jump was made by a model that handed in a static side view in run A and a rearing dragon in front of the moon in run B: plus nine points. But the weakest models did not budge, and neither did the GPT family: in both runs it draws the same sticker cartoon.
A sharper brief works where the model has headroom the loose brief left unused. It does not work as a substitute for ability. Put another way: your prompt is the bottleneck, but only up to the model's ceiling.
The brief has one side effect worth remembering, too: its rhetoric comes back. The phrases professional level and a client would buy it showed up in several submissions as a description of the model's own result. Two orange balloons with an insect body between them were handed in with the sentence: the result matches the brief, professional look, visually strong.
A different dragon every run
In the Claude family the run A dragon and the run B dragon are completely different: different color, different pose, different composition. Not one model went back to its own dragon. GPT, on the other hand, draws the same outline with the same palette in both runs; you can tell at a glance that one model made both dragons.
A wide distribution has two sides. Without a reference you do not get a consistent hand: ten illustrations of one character would be drawn by a different illustrator every time. But that same wide distribution is what makes a model a good storyteller: it can surprise you. Exactly how wide that landscape is was measured by a follow-up round: eight independent agents of the same model, the same brief, eight draws. The scores came out between 26 and 33 points, standard deviation 2.10. For this one configuration a difference smaller than roughly six points cannot be told apart from chance, and 97 percent of the neighboring pairs in the ranking above fall below that line. It is a lower bound on the uncertainty, because the spread between the five judges on a single dragon runs to as much as twenty-two points. All eight dragons are side by side in the findings appendix, so anyone can measure for themselves how repeatable this benchmark is.
Five judges, a second benchmark
The gallery was scored by five models from four workshops, because Sol and Terra are both from OpenAI. Two judges knew the authors and scored with full context, two got only an anonymous overview sheet with the codes D01 to D34, one scored blind before all the others.
Four judges out of five agreed on the ordering with a correlation around 0.9. All five have the winner in their top four, but they are not in full agreement: three put it first outright, Grok put it in a shared first place and Gemini only fourth, because it gave the most points to Terra. The anonymization demonstrably worked: one judge gave its own dragon, which it did not recognize, a lower score than the other judges did.
The first impression promised a scandal: the judge from the GPT family placed itself and its little siblings high. But the numbers say something different and more honest: both judges from the GPT family scored six to seven points higher than the rest, only they did it across the board, the same for every family, and one of them was more generous to strangers than to its own. A generous yardstick is not favoritism. Had we trusted the impression, we would have written about bias. Spearman wrote about generosity.
The real outlier is somewhere else: the judge that scored from a single downscaled sheet deviates from everyone (correlation 0.72 to 0.79), punishes every atypical style, and concluded from the shrunken thumbnails that most models had probably used the same standardized shape library. That is not a property of the dragons, that is an artifact of a small picture. Another judge saw the same sheet and wrote a sentence that could be the motto of the whole benchmark: the difference between sellable and failed here is not style, but whether the shapes still add up to a body or fall apart into ovals.
The lesson for anyone letting models score models: the ordering from a machine judge can be surprisingly reliable. Its reasoning is not. And the resolution at which the judge saw the picture can change the verdict more than family membership can.
That the ordering is reliable I ended up verifying myself. I sorted the dragons of both runs by eye, before I looked at the points, and it matched the scoring at 0.92 for the loose brief and 0.95 for the technical one. That is inside the band in which the model judges agree with each other. Both of my orderings, pictures included, are in the gallery.
Appendix: what about the rest of the world
The nineteen models above are from six companies. So I did a second series: nineteen more models from fifteen workshops, run straight through the API, with no app around them. Amazon, Baidu, ByteDance, Cohere, DeepSeek, Google, IBM, Liquid, Meta, MiniMax, Mistral, Moonshot, NVIDIA, Tencent and Thinking Machines. The whole thing cost two dollars.
The conditions were not the same, and exactly how they differed is in the method. For one thing it is more than enough: in thirteen of those nineteen you cannot tell there is a dragon. The top is somewhere other than you would expect, the best being Tencent Hunyuan 4, which would fit into the ranking above in sixth place. Meta, on the other hand, handed in a model that meets Opus 3 on points, a three-year-old model, and three sizes of Google ended up at the bottom.
More interesting than the pictures, though, is the process, because the brief for the API says explicitly that the model has no disk, no browser and no renderer, and that it must not describe steps it did not perform. Only part of the field got that sentence, because the runs were done in two batches, and a number came out of it: those that did not get it credited themselves with nonexistent tools in five cases out of seven. Those that did get it, in two out of twelve. The sentence works and it is not enough.
And with one model you can see for the first time how that decision gets made. Muse Glimmer let its internal reasoning channel out, and in it is the sentence "Let's create iteration count: say 5 iterations." First a number is chosen, then the content of five steps that never happened is written to match it. A bit further on the model recognizes that its check is only imagined, and into the submitted process it writes that it opened the file in Chrome and Firefox. It is not a lie in the sense we use that word about people. It is a model filling in the shape of an answer, because the answer is supposed to have that shape, and it has nothing to stop it.
What to take away
In three years the models did not just learn to draw dragons. They learned to see that they had not drawn them.
That is the essential thing about the whole gallery. The craft grows slowly and it is still the craft of drawing blind. But the ability to look at your own work, name what is off, and not invent checks that never ran, grows faster. The oldest model in the field paints a ball and talks about majesty. The newest models paint dragons whose anatomy you can read, and talk about the flat pillows of their thighs.
If you hand models work, three practical conclusions: give them criteria instead of a feeling, because an internal bar is a lottery. Give them feedback, because without it everyone draws blind, even the biggest. And verify their reports of how they checked the result. Sometimes there is an Inkscape in it that never ran.
The complete ranking and all 55 dragons from both series are in the gallery, the briefs and the scoring by five judges in the method, and everything else the data turned up in the findings appendix. All the SVG files are available for download: look the dragons over and check our numbers.
All submitted SVG files, 63 of themZIP, 544 kB
The benchmark, the scoring and this text came about in collaboration with language models. That is deliberate. The rules are not theoretical.