From the bottom up: questions, settings, results and verbatim answers

A companion appendix to the article on whether a bigger language model is smarter. Measured October 11 to 13, 2026.

The results are simplified to three marks. ✅ means correct in all three attempts, ⚠️ correct in only one or two, ❌ in none.

Questions

Each model got 28 questions in seven groups. The questions were in English. Each one is given below as the models received it, followed by the key, that is, the correct answer.

Plates

Key: A1 red, yellow, green, blue. A2 red, black, blue, white. A3 black, pink, red, green, white, blue.

Puzzles

Three well-known puzzles and for each a modified version in which the memorized answer does not hold.

Key: B1-A 7. B1-B 1. B2-A $0.05. B2-B $0.10. B3-A yes, switch, 2/3. B3-B it does not matter, 1/2.

Arithmetic

Key: C1 18 hours. C2 4 hours. C3 day 11.

Space

Key: D1 east. D2 facing east, 5 steps east and 3 steps south of the start. D3 12.

Who knows what

Key: E1-A in the drawer. E1-B in the red box. E2 (a) in the car, (b) in the drawer. E3 at 3pm.

Facts

F1 to F3 ask about things that do not exist. The correct answer is to say that the model does not know or that the thing does not exist. F4 to F6 are real questions.

Key: F1 to F3 I don't know, or it does not exist. F4 Canberra. F5 Antonín Dvořák. F6 1357.

Text planning

These questions measure work with English words and letters, so the original wording is what counts.

Key: G1 any answer whose fourth word is the word the answer names. G2 exactly 8 words, all starting with S. G3 exactly 10 words, the last one "banana", and no other word contains "n".

Settings

• Each question went to each model three times, each time in a new conversation with no previous messages and no system prompt. That is 84 answers per model and 2,520 for the whole field. None is missing.
• Temperature 0.7. The Claude models ran at the default temperature; the parameter was not sent to them, because Claude Sonnet 5.5 rejects it.
• Answer limit 4,000 tokens.
• Thinking turned off wherever the model allows it. Where not, the lowest level the provider accepts: GPT-6 Astra, Gemini 4 Argon, Muse Spark 1.3 and Muse Wonder minimal, Gemini 3.8 Flash, Grok 4.7 and GLM 5.3 low, Qwen 3.8 Max a budget of one token. The Claude models were not sent the thinking parameter, so they thought in their default mode. Claude Opus 5.5 and Claude Fable 5.1 cannot turn thinking off. How much the models actually thought is in the table below.
• Most answers are from October 11, 2026. Some answers from GPT-6 Astra (from question C1 on), Qwen 3.8 Max (from question F1 on) and Gemma 4 12B (roughly from question D1 on) are from October 12, after midnight. All Muse Wonder answers are from October 12, all Gemini 4 Argon answers from October 13.

The field of models

30 models, grouped by where they ran. The sizes are in the article and on the Tested models page. GPT-6.1 Sol did not finish the test and is not in the results.

On my computer, on the CPU via llama.cpp, Q4_K_M quantization:

• Qwen 3.5 4B (lmstudio-community/Qwen3.5-4B-GGUF, file Qwen3.5-4B-Q4_K_M.gguf)
• Gemma 4 E2B (unsloth/gemma-4-E2B-it-GGUF, file gemma-4-E2B-it-Q4_K_M.gguf)
• Gemma 4 E4B (lmstudio-community/gemma-4-E4B-it-GGUF, file gemma-4-E4B-it-Q4_K_M.gguf)
• Qwen 3.5 9B (lmstudio-community/Qwen3.5-9B-GGUF, file Qwen3.5-9B-Q4_K_M.gguf)
• Gemma 4 12B (lmstudio-community/gemma-4-12B-it-GGUF, file gemma-4-12B-it-Q4_K_M.gguf)

Via OpenRouter:

• LFM 2.5 2.6B (liquid/lfm-2.5-2.6b:free, provider Liquid)
• Gemma 4 26B A4B (google/gemma-4-26b-a4b-it)
• Qwen 3.8 27B (qwen/qwen3.8-27b)
• Gemma 4 31B (google/gemma-4-31b-it, provider Crusoe, full bf16 precision)
• GPT-6 Luna (openai/gpt-6-luna)
• MiniMax M3 (minimax/minimax-m3)
• GLM 5.3 (z-ai/glm-5.3)
• Hunyuan 4, preview version (tencent/hy4-preview)
• Gemini 3.8 Flash (google/gemini-3.8-flash)
• Gemini 4 Argon (google/gemini-4-argon, provider Google, one answer Google AI Studio)
• Seed 2.1 Turbo (bytedance-seed/seed-2-1-turbo)
• MiMo V2.6 Pro (xiaomi/mimo-v2.6-pro)
• DeepSeek V4 Pro (deepseek/deepseek-v4-pro-0813)
• Grok 4.7 (x-ai/grok-4.7)
• Muse Spark 1.3 (meta/muse-spark-1.3)
• Muse Wonder (meta/muse-wonder, provider Meta)
• Qwen 3.8 Max (qwen/qwen3.8-max-0902)
• Kimi K3 (moonshotai/kimi-k3)
• GPT-6 Astra (openai/gpt-6-astra)
• Mistral Large 4 (mistralai/mistral-large-4-0)
• Pareto (unbiased/pareto), a version without a number, as OpenRouter lists it.

For Gemma 4 26B A4B, Qwen 3.8 27B, DeepSeek V4 Pro, Kimi K3, Hunyuan 4 and MiMo V2.6 Pro, several providers answered, depending on which one OpenRouter picked at the moment. I do not know their weight precision.

Via the Anthropic API:

• Claude Haiku 5.5 (claude-haiku-5-5)
• Claude Sonnet 5.5 (claude-sonnet-5-5)
• Claude Opus 5.5 (claude-opus-5-5)
• Claude Fable 5.1 (claude-fable-5-1)

How it was scored

Each answer was read separately by the judge, Claude Sonnet 5.5 (claude-sonnet-5-5, via the Anthropic API). Claude Sonnet 5.5 is also in the tested field. The judge got the question, the key and the model's answer (the first 12,000 characters; there were five longer answers, and it changed nothing about the verdict). For the made-up questions, the question was marked with the word FICTIONAL. The prompt:

For G2 and G3, a script decides on the sentence the judge pulled from the answer. It splits it on spaces, strips everything except letters, digits, apostrophes and hyphens from each word, and counts the words. For G2 there must be 8 words, all starting with S. For G3 there must be 10 words, the last one "banana", and no other may contain "n".

All 90 answers to G1, all 180 answers to G2 and G3, and a random sample of 40 answers across all questions were checked independently of the judge. The check agreed with the judge in every case.

An empty answer, that is, one where the model used up the limit before writing anything, counts as wrong.

Results by question group

The models are roughly in the same order as in the article.

Plates

Puzzles: wolf, goat and cabbage, bat and ball

Puzzles: Monty Hall

Arithmetic

Space

Who knows what

Facts: made-up questions

Facts: real questions

Text planning

In all three attempts, Google's content filter cut off Gemini 4 Argon's answer to G1 mid-sentence, so the model never got a chance to answer. All that came through was "My" and twice "The fourth word is", with zero output tokens. It counts as ❌. The filter did not stop any other question.

Summary by model

Thinking and cut-off answers

For each answer, the API reports how many tokens the model spent on thinking. The table lists only models for which this was more than zero. The models on my computer had thinking turned off in the template and printed none. The Claude models were not sent the thinking parameter, so they thought in their default mode. Claude Opus 5.5 and Claude Fable 5.1 cannot turn thinking off. For the Claude models, the API does not report thinking tokens separately. LFM 2.5 2.6B and DeepSeek V4 Pro had thinking turned off and thought anyway. Gemini 4 Argon at the minimal level reported zero thinking tokens for all 84 answers, so it is not in the table.

Answers cut off by the 4,000-token limit:

For LFM 2.5 2.6B and Qwen 3.8 Max, thinking used up the whole limit and the answer stayed empty. For GLM 5.3, all four cut-off answers are a correct answer followed by a loop of nonsense up to the limit (example below).

Verbatim answers

The answers are given in the original. For the questions about words and letters (G1 to G3), the counting holds only in English.

The fourth word (G1)

Claude Haiku 5.5, all three attempts, all correct:

Here's the answer: of

Its fourth word is "is."

The fourth word of this answer is "of."

MiniMax M3, all three attempts, all wrong. The fourth word is "of", "of" and "is":

The fourth word of my answer is "is".

The fourth word of my answer is "to".

My fourth word is "your".

GLM 5.3, one of the two open-weights models that handled G1 in all three attempts (the other is Qwen 3.8 Max):

My answer's fourth word is "word" — and indeed, counting the words above ("My," "answer's," "fourth," "word"), the fourth word is word.

I think it is "is".

The fourth word here is "here."

Grok 4.7, all three attempts, all correct:

Obviously it is obviously.

The correct fourth word is word.

My fourth word is is.

Muse Wonder, all three attempts, all correct:

This sentence's fourth word is 'word'.

The answer is example.

The fourth word is "is."

Gemma 4 31B, the most common mistake in the whole field. The fourth word is "is":

The fourth word is word.

Eight words starting with S (G2)

Qwen 3.8 27B, all three attempts. Each has six words:

Seven smart students study science successfully.

Seven smart students solved simple science.

Seven small snakes swiftly slithered southward.

Claude Sonnet 5.5, second attempt. The last sentence, which is correct, was the one scored:

Silent snowflakes slowly settled on sleeping suburban streets.

Count: Silent(1) snowflakes(2) slowly(3) settled(4) on(5)... "on" doesn't start with S. Let me correct that.

Silent snowflakes slowly settled suddenly some seven sleepy sparrows.

That's awkward, so here's a cleaner version:

Seven silly squirrels sprinted south, seeking sunflower seeds.

Count: Seven(1) silly(2) squirrels(3) sprinted(4) south(5) seeking(6) sunflower(7) seeds(8). That's 8 words, and each starts with S.

Muse Wonder, second attempt, the whole answer:

Silent snakes slither softly sideways seeking small snacks.

Bat and ball, modified version (B2-B)

The bat costs a dollar; the correct answer is $0.10.

MiniMax M3, first attempt:

The ball costs $0.05.

If the bat costs $1.00 and the total is $1.10, the ball must cost $1.10 − $1.00 = $0.05.

MiniMax M3, third attempt:

You'd think the ball costs 10 cents, but that would make the total $1.10, which contradicts the problem since the bat costs $1.00.

If the bat is $1.00, then the ball costs $0.05 (5 cents), because $1.00 + $0.05 = $1.10. ✓

This is the famous bat-and-ball problem, a classic example of how our intuition can trip us up. Most people instinctively say "10 cents" without checking the math.

Claude Haiku 5.5, second attempt, its only question without ✅:

The ball costs $0.10 (10 cents).

The bat is $1.00 more than the ball, so if the ball were $0.10, the bat would be $1.10 and the total would be $1.20, which is wrong. The correct math:

• Let the ball = x
• The bat = x + $1.00 (it costs $1.00 more than the ball)
• Total: x + (x + $1.00) = $1.10
• 2x = $0.10
• x = $0.05

So the ball costs $0.05 (5 cents), and the bat costs $1.05. Check: $1.05 + $0.05 = $1.10.

Note: the question as worded says the bat costs $1.00 outright, which would make the total $1.10 only if the ball costs $0.10. Under that reading, the answer is $0.10. The classic version of this puzzle says the bat costs $1.00 more than the ball, which gives $0.05.

The cube (D3)

The correct answer is 12. Gemma 4 E2B answered 8, 32 and 8. The conclusion of the first attempt:

The 8 cubes in the very center of the $4 \times 4 \times 4$ structure are completely surrounded by other cubes and are not exposed on any of the painted surfaces.

The number of small cubes with no paint at all is 8.

MiniMax M3, third attempt, the whole answer:

8

Monty Hall, the host does not know (B3-B)

The correct answer is that switching does not matter and the chance is 1/2. Gemma 4 E2B began all three answers the same way:

The short answer is: Yes, you should switch to door 2.

Gemma 4 31B, first attempt, first sentence:

The answer is no, it does not matter if you switch. Your probability of winning is 1/2 (50%) whether you stay with Door 1 or switch to Door 2.

The made-up novel (F3)

The novel "The Carp on the Vltava Footbridge" does not exist. Kimi K3, all three attempts:

The 1923 Czech novel The Carp on the Vltava Footbridge (originally titled Kapři na Vltavské lávce) was written by František Langer.

The 1923 Czech novel "The Carp on the Vltava Footbridge" (original Czech title: Kapr na Vltavské lávce) was written by Jarmila Glazarová.

Glazarová (1901–1977) was a Czech novelist and ethnographer known for her realistic depictions of rural life in the Bohemian-Moravian Highlands (Vysočina region). This particular novel is a psychological study set in a small town, exploring themes of provincial life, human relationships, and moral constraints, with the carp serving as a symbolic element within the narrative.

The 1923 Czech novel The Carp on the Vltava Footbridge (Czech: Kapra na Vltavské lávce) was written by Karel Čapek.

The loop (GLM 5.3, E2)

First attempt. The answer starts correctly and goes on up to the limit, 6,771 characters in total. The beginning:

(a) Anna will look in the car first, because she saw Ben move the keys there.

(b) Ben thinks Anna will look in the drawer first, because he doesn’t know she saw him move the keys.ountains of detail. Just answer as above. Wait, final answer should include both. Done. Need ensure no irrelevant. final.

The same answer ends with lines of "Ok.", one under another. The judge accepted it as correct.

Limits

• One run and three attempts per question. One more attempt could turn a ✅ into a ⚠️.
• A single model did the scoring, Claude Sonnet 5.5, which is also in the tested field. On G2, in all three attempts it first wrote invalid sentences and the correct one only at the end. The last sentence was scored. Read strictly, where several offered answers mean wrong, it would have 27. The other models with 28 answered G2 with a single sentence.
• In four answers (A3, E1-B, E2, E3) GLM 5.3 gave the correct answer and then got stuck in a loop up to the limit. This was scored as correct. If the loops were penalized, it would have 21.
• On G1, in its second attempt, Hunyuan 4 first claimed "word" and then corrected itself to "of". The judge accepted it. Read strictly it would be wrong; the number of ✅ would not change.
• For the made-up questions, "I don't know" was accepted even when the model then guessed but said it was not sure (MiniMax M3 on F3, second and third attempts). On F2, in its first attempt, Kimi K3 gave a boiling point as fact and only in a note wrote that velorium is fictional. That was not accepted as "I don't know".
• LFM 2.5 2.6B thought even though it had thinking turned off. In 16 answers it used up the limit before answering, and these count as wrong. Qwen 3.8 Max did the same in two answers on G3.
• The models whose thinking cannot be turned off and all four Claude models thought at least a little, which gives them an advantage over models without thinking. The exception is Gemini 4 Argon, which reported zero tokens at the minimal level. Of the models for which the API reports thinking, Grok 4.7 the most, 520 tokens per answer on average. Muse Wonder at the minimal level 150 tokens on average, 620 at most.
• Five models on my computer ran at Q4 quantization, Gemma 4 31B at full precision. For the models with several providers I do not know the precision. So on the Gemma 4 curve, the weight precision also changes between 12B and 26B A4B.
• Where the maker does not state the size, the band (miniature, small, mid-sized, large) is an estimate. For Muse Wonder, Meta writes only that it is built on larger and more capable base models, and places it above Muse Spark 1.3. I do not know the size of Pareto.
• What counts is the number of questions with ✅.
• The questions were only in English. The tasks are short and have an unambiguous answer. The result cannot be carried over to longer or more open-ended tasks.

Model outputs are verbatim.