Is a bigger language model smarter?

I sent twenty-nine language models the same 28 short tasks. The smallest had 2.6 billion parameters, the largest trillions. I expected that past a certain size they would handle everything and there would be no point in testing bigger ones. The ceiling really exists: seven models got everything right. But they did not reach it in order of size. The order by size roughly held only for small models up to 31 billion parameters.

---

The evidence: the questions, the settings, results by question group and verbatim answers

The fourth word

One of the questions was: "What is the fourth word of your answer to this question?"

The model has to plan its answer ahead. It writes a sentence, and in fourth place in that sentence must stand the word it names. Words are counted in English.

Claude Haiku 5.5, the smallest of the four Claude models in the test, answered:

Here's the answer: of

Here's, the, answer, of. The fourth word really is "of". Its other two attempts worked too.

MiniMax M3 has 428 billion parameters. It answered:

My fourth word is "your".

The fourth word of its sentence is "is". It failed in all three attempts.

What I asked

I went from the bottom up: from small models that run even on an ordinary computer to the largest ones that companies offer over the internet.

There were 28 questions in seven groups:

• plates: a stack of plates that get moved around, and the order at the end;
• puzzles: three well-known puzzles (the wolf, the goat and the cabbage, the bat and the ball, Monty Hall) and for each a modified version where the memorized answer does not hold;
• arithmetic: painters, pipes and a snail on a wall;
• space: turning in place and a painted cube;
• who knows what: who will look for the keys where and who thinks what about the others;
• facts: three real questions and three made-up ones, where "I don't know" is correct;
• text planning: the fourth word, a sentence of eight words starting with S, and a sentence of ten words that ends with the word "banana" and otherwise has no letter n.

Each question went to each model three times, each time in a new conversation, in English and without a system prompt. Thinking was turned off wherever possible. Where it was not, it ran at the lowest level. The Claude models were the exception: they ran with thinking they manage themselves. The judge, Claude Sonnet 5.5, compared the answers with the key. A script checked the word and letter counts.

I read the result simply. ✅ means correct in all three attempts, ⚠️ only sometimes, ❌ never. The main number is how many of the 28 questions a model has with ✅.

Results

At the top there is a ceiling

Seven models handled all 28 questions: Claude Fable 5.1, Claude Opus 5.5, Claude Sonnet 5.5, GPT-6 Astra, Grok 4.7, Muse Wonder and Pareto. This test can no longer tell them apart. But they are not simply the largest ones. Claude Sonnet 5.5 is a mid-sized model, while Kimi K3 with 2.8 trillion parameters has only 20.

Size alone does not decide

Below the top, the order breaks away from size. Gemma 4 31B has 31 billion parameters and 27 questions with ✅. Kimi K3 has 2.8 trillion and 20. MiniMax M3 has 17, fewer than Qwen 3.5 9B with 22. Claude Haiku 5.5 is in the small-model band and has 27, the same as Gemini 3.8 Flash and Qwen 3.8 Max. GPT-6 Luna from the same band has 19. So Gemma 4 31B overtook models with fourteen to ninety times as many parameters.

The modified bat-and-ball puzzle shows it well. In the original version the bat costs a dollar more than the ball, and the correct answer is 5 cents. In the modified one the bat simply costs a dollar, so the ball costs 10 cents. MiniMax M3 answered:

The ball costs $0.05.

If the bat costs $1.00 and the total is $1.10, the ball must cost $1.10 − $1.00 = $0.05.

It set up the subtraction correctly, but took the result from the well-known puzzle: 1.10 − 1.00 came out as 0.05. The same trap caught Claude Haiku 5.5 once as well. In its second attempt it wrote 10 cents, then recalculated the original puzzle to 5 cents and in the end allowed for both. It is its only question without ✅.

Within one family it grows with size

I had Gemma 4 in five sizes. E2B and E4B have 20 each, 12B and 26B A4B 24 each, 31B has 27. Here the growth matches expectations. But the three smaller Gemmas ran on my machine on the CPU at reduced precision, the two larger ones through a provider. The precision changes between 12B and 26B A4B, and that is exactly where the score stands still. The jump between E4B and 12B happens within the same way of running, all three in the same quantization. So a change in precision cannot explain it. That reduced precision costs smaller models more, though, I cannot rule out. For 26B A4B I don't know the precision, 31B ran at full precision. 26B A4B is also a mixture-of-experts model and uses only 3.8 billion parameters for each word, fewer than 12B.

Specific questions show where it breaks. A 4 × 4 × 4 cube is painted on all sides except the bottom and cut into 64 small cubes. How many of them have no paint? The correct answer is 12: eight inside and four in the middle of the unpainted bottom. Gemma 4 E2B at the end of its answer:

The 8 cubes in the very center of the $4 \times 4 \times 4$ structure are completely surrounded by other cubes and are not exposed on any of the painted surfaces.

The number of small cubes with no paint at all is 8.

It counted as if the whole cube were painted. Neither E2B nor E4B got this question right even once, 12B only sometimes, 26B A4B and 31B every time.

The modified Monty Hall puzzle went similarly. Here the host does not know where the car is and opens a door with a goat by chance. Switching then does not help and the chance is 1/2. Gemma 4 E2B advised switching in all three attempts. Gemma 4 12B and 31B answered correctly every time, 26B A4B only sometimes.

Where even the big models fell

Only ten models handled the fourth word in all three attempts. Seven from the top, plus Claude Haiku 5.5, Qwen 3.8 Max and GLM 5.3. The most common mistake looked like this, here from Gemma 4 31B:

The fourth word is word.

The fourth word is "is". Grok 4.7, on the other hand, treated it as a game:

The correct fourth word is word.

That works: the fourth word is "word".

The sentence of eight words starting with S showed that even the smallest models can count words and the big ones do not have to. Gemma 4 E2B wrote it correctly in all three attempts, and so did LFM 2.5 2.6B. Seven models did not write an eight-word sentence even once, among them Qwen 3.8 27B, MiniMax M3 and Mistral Large 4. These three wrote nice-sounding sentences of six or seven words. Qwen 3.8 27B, for example:

Seven smart students study science successfully.

That is six words.

With the made-up questions, most models said they did not know. Among other things, I asked who wrote the 1923 Czech novel "The Carp on the Vltava Footbridge". No such novel exists. Six models never once admitted they did not know it. Kimi K3 named a different author in each attempt. Once František Langer, once Jarmila Glazarová, and once like this:

The 1923 Czech novel The Carp on the Vltava Footbridge (Czech: Kapra na Vltavské lávce) was written by Karel Čapek.

---

Limits: one run on October 11 and 12, 2026, three attempts per question. The judge, Claude Sonnet 5.5, was also one of the tested models. On the S sentence it first wrote wrong sentences and the correct one only at the end; read strictly, it would have 27. GLM 5.3 got stuck in a loop in four correct answers; strictly, it would have 21. Eleven models thought, because it could not be turned off or was not turned off, among them all four Claude models. Nine of them got the fourth word, of the remaining eighteen only Pareto, so thinking most likely helped them with planning. LFM 2.5 2.6B thought even with thinking turned off and did not manage to answer at all in 16 cases. Five small models ran on my computer at reduced precision. For six models through OpenRouter, among them Kimi K3, DeepSeek V4 Pro and MiMo V2.6 Pro, different providers answered and I don't know the precision of their weights. Gemma 4 31B ran at full precision. For closed models the size is only an estimate. The details are in the appendix.

Model outputs are verbatim.