Skip to content
EshAlora · AI Behavior Lab
Česky English

Do AI models agree when they review the same manuscript?

I had one manuscript read by five models and expected five versions of the same review. I got five different books.


A fantasy manuscript of about 83,000 words. The smallest model, Claude Haiku 4.5, wrote that it is "a story of emancipation about self-discovery, manipulation, and personal power." For Claude Sonnet 5 it was "a readable, technically solid young-adult fantasy" in which "a travel romance turns into a political thriller."

The readers were four models from the Claude family. Besides Haiku 4.5 and Sonnet 5, they were Claude Opus 4.8 and Claude Fable 5. All of them read in early July 2026; Sonnet 5 had come out two days earlier, and I already had it in the app. According to the Tested models page, Haiku 4.5 is small, Sonnet 5 medium, and Opus 4.8 and Fable 5 large. The maker does not publish sizes; the bands are this site's estimate. When I write smallest and largest, I mean the order in the lineup and in price: through the API, the cheapest of the four was Haiku 4.5, the most expensive Fable 5. Each read the book separately and once, under the same protocol. Fable 5 read first, in the thread where the book had been worked on before. Whether it got only the book and the protocol, like the other three, I don't know. And that the reader really was Fable 5, I know only indirectly: the reading drew on its own usage limit. The fifth was ChatGPT, reading freely, without a protocol. Its version was not recorded. I had its report on July 2, 2026, so it was GPT-5.5 at most.

Because each read only once, I don't know how much of the differences below comes from the model and how much from the chance of a single reading.

Each model read a different book

Opus 4.8 read like a dramaturge. It followed the curve of stakes and tension and wrote that "the reader soon stops being afraid for the heroine." Fable 5 kept a ledger of debts. It wrote down what the text had promised itself and never paid back: contradictions, guns that never went off, emotional arrears.

ChatGPT behaved like a developmental editor. For every problem it went straight to prescribing the treatment, including ready-made replacement sentences and a thesis about what the book is really about. But it was the only one that did not get the protocol. So part of that difference may be down to the different prompt.

Whatever the summary went after, the criticisms went after too. Haiku 4.5 came back to morality and unprocessed emotions more often than the others. Sonnet 5 named pacing and exposition as the main weakness. With human beta readers I know it from experience: a reader builds a frame, and that frame then decides what they see in the book. With the models, on this book, it looked the same. Whether a model built its frame in the first chapters or only while writing the report is something a single reading cannot tell you.

Agreement at just two places

I had a hypothesis that the findings would nest like a matryoshka. Whatever the small model finds, the large one finds too, plus something extra. What came out was more like a flower: a small shared core and large petals of their own.

The four readers who followed the protocol reported 135 items in total. Absolutely all of them agreed on two places. One is a numerical inconsistency in the heroine's age. The other is a sudden turn by one character in an ethically difficult scene. At four more places, three of them met. For the three smaller models, roughly half of the criticisms overlapped with at least one other reader. For Fable 5 it was only 14 of 71 criticisms; it did not put positives in its table. It is not down to a finer split of the findings: according to the merge, Fable 5 split someone else's criticism into two of its own only twice, and twice it did the opposite and covered two of theirs with one. And within the trio of smaller models, the smallest one brought the most items. The matches were paired by one of the readers, see Limits.

When I add ChatGPT, a single shared place remains: that ethically difficult scene. ChatGPT did not report the age inconsistency. Five readers from two model families stopped at the same scene. But they did not agree on what is wrong with it. Four write that the text dealt with its consequences too quickly. Opus 4.8 only noticed that the character's turn came suddenly, and gave it the lowest severity. About how the text handled the scene it wrote: "The aside handles it responsibly and maturely". A place like that is a candidate for priority number one, even if the readers do not agree on what exactly is wrong with it.

When the readers contradict each other

In several places the readers directly contradicted each other. Take the revelation at the end of the book. Opus 4.8 writes that it arrives so densely packed that "the reader has no time to absorb it," and advises to "hint at the goddesses' falseness earlier." Haiku 4.5 put it among the positives: "the clue pays off. Cleanly planted."

The heroine's aliases are similar. For Haiku 4.5 they are a "confusing naming burden" it suggests trimming. For Opus 4.8 they are literally "praise, not a criticism." The runes that appear at the end as a new rule are seen by both Haiku 4.5 and Fable 5 as a solution pulled out of a hat, and both want them planted earlier. Opus 4.8, on the contrary, cites them as a good example of the price the heroine pays for her power.

With human beta readers this would surprise no one; the audience is not a monolith. I read it the same way with the models. I take them as four audience segments, not as four failed attempts at one correct reader. Where they contradict each other, your readership splits, and you have to choose who you are tuning the scene for. The caveat of a single reading applies here too. I don't know whether a model would not contradict itself on a second reading.

How to add up the reports

When you merge reviews, three traps are waiting that will eat your whole experiment.

First: counts of criticisms are not comparable without adjustment. The most diligent of the smaller readers was Haiku 4.5, but 27 % of its items were positives. Those are not errors but a list of places not to touch. In my view that is a valuable output, even though the protocol explicitly did not want praise. Once the positives are subtracted, Haiku 4.5 leads Opus 4.8 by only three criticisms.

Second: each model grades differently. Haiku 4.5 graded fifteen of its 22 criticisms "moderate." Sonnet 5 graded nine of thirteen "minor." Sonnet 5 has the same inconsistency in the heroine's age as critical, the other three as moderate. Before merging, you have to regrade everything on a single scale.

Third: take symptoms from your readers, not cures. A proposed fix may collide with facts of the saga that the reader has not seen. A symptom such as "the numbers don't add up" or "I got lost here" is reliable. The cure is worked out again, from the canonical facts.

Who found what

Other predictions were stated in advance too, mine and those of Fable 5. The expectation was that small models would report more false findings. But the smallest reader's criticisms were, with two exceptions, factual and anchored to chapters; I read all of them. For the others I did not count false findings. I expected ChatGPT to slack off. Fable 5 expected fewer chapter references from it, grades, and a frame of its own forced on the book. But ChatGPT delivered what I think was the most orderly editorial document of the five, with priorities A to C and not a single score. The first of those five predictions I checked only for the smallest reader. Of the other four, only the last came half true: it gave the book a thesis of its own.

Only one thing survived, in a narrower form, and even that with the caveat that the largest model read in the thread where the book had been worked on: among the Claudes, inconspicuous long-range dependencies were found only by it. They include a vanished artifact, an overview of which characters know the heroine's secret, and a contradiction in one word across seven chapters. But ChatGPT caught that contradiction too. The conspicuous inconsistency in the heroine's age, on the other hand, was caught by all four Claudes. On this book it looks like what decides is not the distance between the places, but how conspicuous the anchor is.

On this book there was no best beta reader. Fable 5 reported the most criticisms, and yet it missed what Haiku 4.5 found. For example, that the heroine "refuses to kill, 30 lines later 'Kill them all.' The transition is missing." Haiku 4.5 and Sonnet 5 were also cheaper through the API than the large models. In my view, adding them to one large reading does not cost much. Together, the four readers found more than any one of them alone.


Limits: the reports were compared by Claude Fable 5, itself one of the readers. The matching criticisms were also merged, independently of it, by Claude Opus 4.8, when it built a list of fixes from the reports. It is among the readers too. The agreement figures are calculated from Claude Opus 4.8's merge. Both comparisons found the same two places where all four met. I also read the four Claude reports myself; the counts and quotes are taken directly from the reports.

Model outputs are translated; the content is unchanged.